CN110399380A - A kind of data processing method, electronic device and storage medium - Google Patents
A kind of data processing method, electronic device and storage medium Download PDFInfo
- Publication number
- CN110399380A CN110399380A CN201910525181.2A CN201910525181A CN110399380A CN 110399380 A CN110399380 A CN 110399380A CN 201910525181 A CN201910525181 A CN 201910525181A CN 110399380 A CN110399380 A CN 110399380A
- Authority
- CN
- China
- Prior art keywords
- data
- sublist
- field
- filled
- identification information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 16
- 241001269238 Data Species 0.000 claims abstract description 16
- 238000013507 mapping Methods 0.000 claims abstract description 12
- 238000000034 method Methods 0.000 claims abstract description 9
- 238000012545 processing Methods 0.000 claims abstract description 9
- 238000007689 inspection Methods 0.000 claims description 26
- 238000012795 verification Methods 0.000 claims description 8
- 238000004140 cleaning Methods 0.000 claims description 2
- 230000008859 change Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000012937 correction Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention relates to technical field of data processing, provide a kind of data processing method, electronic device and storage medium, this method obtains more pending datas in the first preset time from database, according to the mapping relations of pre-set each sublist and data identification information, data identification information corresponding with each sublist is found out in pending data, carries out data cleansing after the corresponding pending data of data identification information is filled into corresponding sublist.Later, the sublist for completing data cleansing is saved in database；After receiving data summarization instruction, from all sublists read in the second preset time in database, according to the corresponding relationship of different field record area and sublist in wide table, the data that the sublist of reading records are updated to the corresponding field record region of wide table.The data that the present invention generates each timing node periodically carry out statistical updating using unified form template, it can be ensured that the consistency of data statistics.
Description
Technical field
The present invention relates to technical field of data processing more particularly to a kind of data processing method, electronic device and storage to be situated between
Matter.
Background technique
Currently, actuarial is measured (Actuarial measurement) and financial calculations by domestic many companies
(Financial calculation) carries out independent statistics.Actuarial metering counts the project of various demands, and finance are right
The Various types of data of disengaging carries out list statistics, and the data processed result of the two can reflect respective statistical result.
However, usually causing to come out since the data bore of acquisition is different or timing node disunity when statistics
Result have error, it is difficult to guarantee the consistency of data counted.
Summary of the invention
The purpose of the present invention is to provide a kind of data processing method, electronic device and storage mediums, it is intended to pass through acquisition
The data that each timing node generates, periodically carry out statistical updating, it is ensured that the consistency of data improves the accuracy of statistics.
To achieve the above object, the present invention provides a kind of data processing method, this method comprises:
Obtaining step: more pending datas in the first preset time, every pending data packet are obtained from database
Include data identification information；
Finding step: each sublist and data identification information in multiple sublists for including according to pre-set wide table
Mapping relations find out data identification information corresponding with each sublist in the pending data；
Filling step: according to the data identification information corresponding with each sublist found out from the pending data,
The corresponding pending data of the data identification information is filled into corresponding sublist；
Verification step: data cleansing is carried out to the filled sublist, the sublist for completing data cleansing is saved in institute
State database；
It updates step: receiving data summarization instruction, read this from the database and summarize specified second default of instruction
All sublists in time, according to the corresponding relationship of different field record area and sublist in the wide table, by the sublist of reading
The data of record are updated to the corresponding field record region of the wide table.
Preferably, the foundation of the sublist includes:
The wide table is divided into a series of field record regions；
According to the field attribute in each field record region, all fields in each field record region divide
The each set of fields for including to each field record region；And
Each set of fields that each field record region includes is arranged to corresponding sublist.
Preferably, the filling of the sublist includes:
According to the field attribute of the sublist, all fields of the sublist are divided, obtaining the sublist includes
Each field set；
Judge the corresponding pending data of the data identification information whether be in sublist needed for the field value of field to
Handle data；
If so, determining the fields match of the pending data and corresponding sublist, and it is filled into corresponding field in sublist
Region.
Preferably, described to include: to the filled sublist progress data cleansing
Consistency check is carried out to the data in the filled sublist, one including the data in the single sublist
The consistency check of data between cause property inspection and each sublist；
Invalid value inspection is carried out to the data in the filled sublist；And
Missing values inspection is carried out to the data in the filled sublist.
Preferably, the update step includes:
The pending data filled in sublist is added in wide table accordingly by the corresponding relationship in field record region and sublist
Field record region be updated；
Summarize field record region according in wide table, associated pending data is read from multiple sublists and carries out data
Summarize, by the obtained data summarization be added in wide table described in summarize field record region and be updated.
In addition, to achieve the above object, the present invention also provides a kind of electronic device, which includes memory and place
Device is managed, is stored with the data processor that can be run on the processor, the data processor quilt in the memory
The processor realizes following steps when executing:
Obtaining step: more pending datas in the first preset time, every pending data packet are obtained from database
Include data identification information；
Finding step: each sublist and data identification information in multiple sublists for including according to pre-set wide table
Mapping relations find out data identification information corresponding with each sublist in the pending data；
Filling step: according to the data identification information corresponding with each sublist found out from the pending data,
The corresponding pending data of the data identification information is filled into corresponding sublist；
Verification step: data cleansing is carried out to the filled sublist, the sublist for completing data cleansing is saved in institute
State database；
It updates step: receiving data summarization instruction, read this from the database and summarize specified second default of instruction
All sublists in time, according to the corresponding relationship of different field record area and sublist in the wide table, by the sublist of reading
The data of record are updated to the corresponding field record region of the wide table.
Preferably, the foundation of the sublist includes:
The wide table is divided into a series of field record regions；
According to the field attribute in each field record region, all fields in each field record region divide
The each set of fields for including to each field record region；And
Each set of fields that each field record region includes is arranged to corresponding sublist.
Preferably, the filling of the sublist includes:
According to the field attribute of the sublist, all fields of the sublist are divided, obtaining the sublist includes
Each field set；
Judge the corresponding pending data of the data identification information whether be in sublist needed for the field value of field to
Handle data；
If so, determining the fields match of the pending data and corresponding sublist, and it is filled into corresponding field in sublist
Region.
Preferably, described to include: to the filled sublist progress data cleansing
Consistency check is carried out to the data in the filled sublist, one including the data in the single sublist
The consistency check of data between cause property inspection and each sublist；
Invalid value inspection is carried out to the data in the filled sublist；And
Missing values inspection is carried out to the data in the filled sublist.
In addition, to achieve the above object, it is described computer-readable the present invention also provides a kind of computer readable storage medium
It include data processor in storage medium, it can be achieved that any one as above when the data processor is executed by processor
The data processing method.
The present invention is by obtaining more pending datas in the first preset time from database, according to pre-set each
The mapping relations of a sublist and data identification information find out Data Identification letter corresponding with each sublist in pending data
Breath, carries out data cleansing after the corresponding pending data of data identification information is filled into corresponding sublist.Later, number will be completed
Database is saved according to the sublist of cleaning；After receiving data summarization instruction, from being read in database in the second preset time
All sublists, according to the corresponding relationship of different field record area and sublist in wide table, data that the sublist of reading is recorded
It is updated to the corresponding field record region of wide table.The data that the present invention generates each timing node utilize unified table mould
Plate periodically carries out statistical updating, it can be ensured that the consistency of data statistics.
Detailed description of the invention
Fig. 1 is the schematic diagram of one embodiment of electronic device of the present invention；
Fig. 2 is the Program modual graph of the data processor preferred embodiment in Fig. 1；
Fig. 3 is the flow chart of data processing method preferred embodiment of the present invention；
The embodiments will be further described with reference to the accompanying drawings for the realization, the function and the advantages of the object of the present invention.
Specific embodiment
In order to make the objectives, technical solutions, and advantages of the present invention clearer, with reference to the accompanying drawings and embodiments, right
The present invention is further elaborated.It should be appreciated that described herein, specific examples are only used to explain the present invention, not
For limiting the present invention.Based on the embodiments of the present invention, those of ordinary skill in the art are not before making creative work
Every other embodiment obtained is put, shall fall within the protection scope of the present invention.
It should be noted that the description for being related to " first ", " second " etc. in the present invention is used for description purposes only, and cannot
It is interpreted as its relative importance of indication or suggestion or implicitly indicates the quantity of indicated technical characteristic.Define as a result, " the
One ", the feature of " second " can explicitly or implicitly include at least one of the features.In addition, the skill between each embodiment
Art scheme can be combined with each other, but must be based on can be realized by those of ordinary skill in the art, when technical solution
Will be understood that the combination of this technical solution is not present in conjunction with there is conflicting or cannot achieve when, also not the present invention claims
Protection scope within.
As shown in Figure 1, being the schematic diagram of one embodiment of electronic device of the present invention.Electronic device 1 is that one kind can be according to thing
The instruction for first setting or storing, the automatic equipment for carrying out numerical value calculating and/or information processing.The electronic device 1 can be
Computer, be also possible to single network server, multiple network servers composition server group or based on cloud computing by
The cloud that a large amount of hosts or network server are constituted, wherein cloud computing is one kind of distributed computing, by a group loose couplings
One super virtual computer of computer set composition.
In the present embodiment, electronic device 1 may include, but be not limited only to, and can be in communication with each other connection by system bus
Memory 11, processor 12, network interface 13, memory 11 are stored with the data processor that can be run on the processor 12
10.It should be pointed out that Fig. 1 illustrates only the electronic device 1 with component 11-13 it should be appreciated that be not required for
Implement all components shown, the implementation that can be substituted is more or less component.
Wherein, memory 11 includes the readable storage medium storing program for executing of memory and at least one type.Inside save as the fortune of electronic device 1
Row provides caching；Readable storage medium storing program for executing can be for if flash memory, hard disk, multimedia card, card-type memory are (for example, SD or DX memory
Deng), random access storage device (RAM), static random-access memory (SRAM), read-only memory (ROM), electric erasable can compile
Journey read-only memory (EEPROM), programmable read only memory (PROM), magnetic storage, disk, CD etc. it is non-volatile
Storage medium.In some embodiments, readable storage medium storing program for executing can be the internal storage unit of electronic device 1, such as the electronics
The hard disk of device 1；In further embodiments, the external storage which is also possible to electronic device 1 is set
Plug-in type hard disk that is standby, such as being equipped on electronic device 1, intelligent memory card (Smart Media Card, SMC), secure digital
(Secure Digital, SD) card, flash card (Flash Card) etc..In the present embodiment, the readable storage medium storing program for executing of memory 11
Commonly used in storing in the operating system and types of applications software that are installed on electronic device 1, such as storage one embodiment of the invention
Data processor 10 etc..It has exported or will export each in addition, memory 11 can be also used for temporarily storing
Class data.
The processor 12 can be in some embodiments central processing unit (Central Processing Unit,
CPU), controller, microcontroller, microprocessor or other data processing chips.The processor 12 is commonly used in the control electricity
The overall operation of sub-device 1, such as execute control relevant to other equipment progress data interaction or communication and processing
Deng.In the present embodiment, the processor 12 is for running the program code stored in the memory 11 or processing data, example
Such as operation data processing routine 10.
The network interface 13 may include radio network interface or wired network interface, which is commonly used in
Communication connection is established between the electronic device 1 and other electronic equipments.
The data processor 10 is stored in memory 11, including the computer-readable finger being stored in memory 11
It enables, which can be executed by processor 12, the method to realize each embodiment of the application.
In one embodiment, following steps are realized when above-mentioned data processor 10 is executed by the processor 12:
Obtaining step: more pending datas in the first preset time, every pending data packet are obtained from database
Include data identification information.
In the present embodiment, periodically obtains in the first preset time (for example, T to T-1) and generate from database
Full dose data.Its mode obtained includes carrying out snapshot to the full dose data in the first preset time, and the snapshot refers to pair
Sometime the data on node are copied.After obtaining full dose data, several data are filtered out from full dose data and are made
For pending data.
Finding step: each sublist and data identification information in multiple sublists for including according to pre-set wide table
Mapping relations find out data identification information corresponding with each sublist in the pending data.
The width table includes multiple sublists, in the present embodiment, wide table be include data forms that multiple association sublists are constituted.
Each sublist fills associated pending data according to field attribute is corresponding, it is ensured that data consistency.
For example, a kind of list of declaration form, including each item is (for example, personal policy information, the premium amount of money or amount for which loss settled
Deng) data, the daily various data that generate or list in data summarization.
In one embodiment, to including different data identification informations in the pending data of acquisition, according to pre-
The mapping relations of each sublist and data identification information that first determine, the different data mark for including from the pending data
Data identification information corresponding with sublist is filtered out in information.
Further, the foundation of the sublist includes:
The wide table is divided into a series of field record regions；
According to the field attribute in each field record region, all fields in each field record region divide
The each set of fields for including to each field record region；And
Each set of fields that each field record region includes is arranged to corresponding sublist.
In the present embodiment, wide table is divided into a series of field record regions, according to field attribute by each field record
The set of fields that region division goes out is being arranged to corresponding sublist.Wherein, the corresponding sublist in a field record region.
Filling step: according to the data identification information corresponding with each sublist found out from the pending data,
The corresponding pending data of the data identification information is filled into corresponding sublist.
Specifically, being looked into after the data identification information corresponding with each sublist found out in the pending data
Pending data where the data identification information found out corresponds to corresponding sublist.
Further, the filling of the sublist includes:
According to the field attribute of the sublist, all fields of the sublist are divided, obtaining the sublist includes
Each field set；
Judge the corresponding pending data of the data identification information whether be in sublist needed for the field value of field to
Handle data；
If so, determining the fields match of the pending data and corresponding sublist, and it is filled into corresponding field in sublist
Region.
Wherein, when field is character type (for example, address name, declaration form state or change reason etc.), to determine with
The pending data of the fields match of corresponding sublist, directly using the pending data as the field value of the field of correspondence sublist into
Row filling；
When field is numeric type (for example, the premium amount of money or indemnity amount of money), to determining and the field of corresponding sublist
The pending data matched directly is filled the pending data as the field value of the field of corresponding sublist, or default
The more pending datas with the fields match of corresponding sublist are determined in time, after more pending datas are summarized as pair
The field value of the field of sublist is answered to be filled.
For example, one field of sublist is (for example, the same day) to collect premium summation in preset time, according to determining and the word
The associated more pending datas (for example, premium M of insurance kind A, the premium N of insurance kind B) of section are summarized, and premium summation (M is obtained
+ N) as corresponding sublist field field value.
Verification step: data cleansing is carried out to the filled sublist, the sublist for completing data cleansing is saved in institute
State database.
The data cleansing refers to that the data identification information to sublist filling is examined and verified, by data cleansing,
It deletes duplicate data identification information or corrects existing mistake, it is ensured that the consistency of data.
Further, described to include: to the filled sublist progress data cleansing
Consistency check is carried out to the data in the filled sublist, one including the data in the single sublist
The consistency check of data between cause property inspection and each sublist；
Invalid value inspection is carried out to the data in the filled sublist；And
Missing values inspection is carried out to the data in the filled sublist.
The consistency check (consistency check) is the reasonable value range and mutual pass according to each data
System checks whether data exceed the range of setting, and logical relation is whether meet the requirement or whether data are conflicting etc..
The invalid value inspection (Invalid Value Check) refers to the whether wrong inspection of the data of filling.
The missing values inspection (Missing Value Check) refers to the inspection that data whether are filled into sublist.
It updates step: receiving data summarization instruction, read this from the database and summarize specified second default of instruction
All sublists in time, according to the corresponding relationship of different field record area and sublist in the wide table, by the sublist of reading
The data of record are updated to the corresponding field record region of the wide table.
In the present embodiment, second preset time is sum time, i.e., what is generated in described second preset time is all
The summation of the sublist of first preset time.For example, second preset time can be one month, a season or 1 year.
As shown in Fig. 2, for the Program modual graph of the data processor preferred embodiment in Fig. 1.
In one embodiment, the data processor 10 includes: to obtain module 101, searching module 102, fill mould
Block 103, correction verification module 104, update module 105.The functions or operations step that the module 101-105 is realized please join above-mentioned figure
The explanation for each step that 1 part is realized when being executed about data processor 10 by the processor 12, and will not be described here in detail.Show
Example property, such as wherein:
Module 101 is obtained, for obtaining more pending datas in the first preset time from database, every to be processed
Data include data identification information；
Searching module 102, each sublist and data mark in multiple sublists for including according to pre-set wide table
The mapping relations for knowing information, find out data identification information corresponding with each sublist in the pending data；
Module 103 is filled, for according to the data mark corresponding with each sublist found out from the pending data
Know information, the corresponding pending data of the data identification information is filled into corresponding sublist；
Correction verification module 104 protects the sublist for completing data cleansing for carrying out data cleansing to the filled sublist
It is stored to the database；
Update module 105, summarizes instruction for receiving data, reads this from the database and summarizes specified the of instruction
All sublists in two preset times will be read according to the corresponding relationship of different field record area and sublist in the wide table
The data of sublist record be updated to the wide corresponding field record region of table.
As shown in figure 3, being the flow chart of data processing method preferred embodiment of the present invention.The present embodiment is at a kind of data
Reason method, this method comprises:
Step S210: more pending datas in the first preset time, every pending data packet are obtained from database
Include data identification information.
In the present embodiment, periodically obtains in the first preset time (for example, T to T-1) and generate from database
Full dose data.Its mode obtained includes carrying out snapshot to the full dose data in the first preset time, and the snapshot refers to pair
Sometime the data on node are copied.After obtaining full dose data, several data are filtered out from full dose data and are made
For pending data.
Step S220: each sublist and data identification information in multiple sublists for including according to pre-set wide table
Mapping relations find out data identification information corresponding with each sublist in the pending data.
The width table includes multiple sublists, in the present embodiment, wide table be include data forms that multiple association sublists are constituted.
Each sublist fills associated pending data according to field attribute is corresponding, it is ensured that data consistency.
For example, a kind of list of declaration form, including each item is (for example, personal policy information, the premium amount of money or amount for which loss settled
Deng) data, the daily various data that generate or list in data summarization.
In one embodiment, to including different data identification informations in the pending data of acquisition, according to pre-
The mapping relations of each sublist and data identification information that first determine, the different data mark for including from the pending data
Data identification information corresponding with sublist is filtered out in information.
Further, the foundation of the sublist includes:
The wide table is divided into a series of field record regions；
According to the field attribute in each field record region, all fields in each field record region divide
The each set of fields for including to each field record region；And
Each set of fields that each field record region includes is arranged to corresponding sublist.
In the present embodiment, wide table is divided into a series of field record regions, according to field attribute by each field record
The set of fields that region division goes out is being arranged to corresponding sublist.Wherein, the corresponding sublist in a field record region.
Step S230: according to the data identification information corresponding with each sublist found out from the pending data,
The corresponding pending data of the data identification information is filled into corresponding sublist.
Specifically, being looked into after the data identification information corresponding with each sublist found out in the pending data
Pending data where the data identification information found out corresponds to corresponding sublist.
Further, the filling of the sublist includes:
According to the field attribute of the sublist, all fields of the sublist are divided, obtaining the sublist includes
Each field set；
Judge the corresponding pending data of the data identification information whether be in sublist needed for the field value of field to
Handle data；
If so, determining the fields match of the pending data and corresponding sublist, and it is filled into corresponding field in sublist
Region.
Wherein, when field is character type (for example, address name, declaration form state or change reason etc.), to determine with
The pending data of the fields match of corresponding sublist, directly using the pending data as the field value of the field of correspondence sublist into
Row filling；
When field is numeric type (for example, the premium amount of money or indemnity amount of money), to determining and the field of corresponding sublist
The pending data matched directly is filled the pending data as the field value of the field of corresponding sublist, or default
The more pending datas with the fields match of corresponding sublist are determined in time, after more pending datas are summarized as pair
The field value of the field of sublist is answered to be filled.
For example, one field of sublist is (for example, the same day) to collect premium summation in preset time, according to determining and the word
The associated more pending datas (for example, premium M of insurance kind A, the premium N of insurance kind B) of section are summarized, and premium summation (M is obtained
+ N) as corresponding sublist field field value.
Step S240: data cleansing is carried out to the filled sublist, the sublist for completing data cleansing is saved in institute
State database.
The data cleansing refers to that the data identification information to sublist filling is examined and verified, by data cleansing,
It deletes duplicate data identification information or corrects existing mistake, it is ensured that the consistency of data.
Further, described to include: to the filled sublist progress data cleansing
Consistency check is carried out to the data in the filled sublist, one including the data in the single sublist
The consistency check of data between cause property inspection and each sublist；
Invalid value inspection is carried out to the data in the filled sublist；And
Missing values inspection is carried out to the data in the filled sublist.
The consistency check (consistency check) is the reasonable value range and mutual pass according to each data
System checks whether data exceed the range of setting, and logical relation is whether meet the requirement or whether data are conflicting etc..
The invalid value inspection (Invalid Value Check) refers to the whether wrong inspection of the data of filling.
The missing values inspection (Missing Value Check) refers to the inspection that data whether are filled into sublist.
Step S250: receiving data summarization instruction, reads this from the database and summarizes specified second default of instruction
All sublists in time, according to the corresponding relationship of different field record area and sublist in the wide table, by the sublist of reading
The data of record are updated to the corresponding field record region of the wide table.
In the present embodiment, second preset time is sum time, i.e., what is generated in described second preset time is all
The summation of the sublist of first preset time.For example, second preset time can be one month, a season or 1 year.
Further, the step S250 includes:
The pending data filled in sublist is added in wide table accordingly by the corresponding relationship in field record region and sublist
Field record region be updated；
Summarize field record region according in wide table, associated pending data is read from multiple sublists and carries out data
Summarize, by the obtained data summarization be added in wide table described in summarize field record region and be updated.
In one embodiment, the wide table is divided into a series of field record regions in advance.
It, will when field record region is the first field type (for example, address name, declaration form state or change reason etc.)
Sublist corresponding with field record region is directly updated to wide table；
When field record region is the second field type (for example, of that month paid premium amount of money or of that month compensation amount of money etc.)
When, associated multiple sublists corresponding with field record region are subjected to data summarization, the data after summarizing are updated to wide table pair
The field record region answered.
When field record region is third field type (for example, the end of month last month premium amount of money, the initial stage premium amount of money or initial stage
Amount for which loss settled etc.) when, another field record region in wide table is updated according to the field of the data summarization recorded in wide table.
It should be appreciated that the type in above-mentioned field record region includes but is not limited to the first field type, the second field class
Type and third field type.
In addition, including in the computer readable storage medium the present invention also provides a kind of computer readable storage medium
Data processor, it can be achieved that following operation when the data processor is executed by processor:
Obtaining step: more pending datas in the first preset time, every pending data packet are obtained from database
Include data identification information；
Finding step: each sublist and data identification information in multiple sublists for including according to pre-set wide table
Mapping relations find out data identification information corresponding with each sublist in the pending data；
Filling step: according to the data identification information corresponding with each sublist found out from the pending data,
The corresponding pending data of the data identification information is filled into corresponding sublist；
Verification step: data cleansing is carried out to the filled sublist, the sublist for completing data cleansing is saved in institute
State database；
It updates step: receiving data summarization instruction, read this from the database and summarize specified second default of instruction
All sublists in time, according to the corresponding relationship of different field record area and sublist in the wide table, by the sublist of reading
The data of record are updated to the corresponding field record region of the wide table.
Computer readable storage medium specific embodiment of the present invention and a kind of above-mentioned data processing method and electronic device
Each embodiment is essentially identical, does not make tired state herein.
The serial number of the above embodiments of the invention is only for description, does not represent the advantages or disadvantages of the embodiments.
Through the above description of the embodiments, those skilled in the art can be understood that above-described embodiment side
Method can be realized by means of software and necessary general hardware platform, naturally it is also possible to by hardware, but in many cases
The former is more preferably embodiment.Based on this understanding, technical solution of the present invention substantially in other words does the prior art
The part contributed out can be embodied in the form of software products, which is stored in a storage medium
In (such as ROM/RAM, magnetic disk, CD), including some instructions are used so that a terminal device (can be mobile phone, computer, clothes
Business device, air conditioner or the network equipment etc.) execute method described in each embodiment of the present invention.
The above is only a preferred embodiment of the present invention, is not intended to limit the scope of the invention, all to utilize this hair
Equivalent structure or equivalent flow shift made by bright specification and accompanying drawing content is applied directly or indirectly in other relevant skills
Art field, is included within the scope of the present invention.
Claims (10)
1. a kind of data processing method, which is characterized in that this method comprises:
Obtaining step: more pending datas in the first preset time are obtained from database, every pending data includes number
According to identification information；
Finding step: the mapping for each sublist and data identification information in multiple sublists for including according to pre-set wide table
Relationship finds out data identification information corresponding with each sublist in the pending data；
Filling step: according to the data identification information corresponding with each sublist found out from the pending data, by institute
It states the corresponding pending data of data identification information and is filled into corresponding sublist；
Verification step: data cleansing is carried out to the filled sublist, the sublist for completing data cleansing is saved in the number
According to library；
It updates step: receiving data summarization instruction, read this from the database and summarize the second specified preset time of instruction
Interior all sublists record the sublist of reading according to the corresponding relationship of different field record area and sublist in the wide table
Data be updated to the wide corresponding field record region of table.
2. data processing method as described in claim 1, which is characterized in that the foundation of the sublist includes:
The wide table is divided into a series of field record regions；
According to the field attribute in each field record region, all fields in each field record region are divided to obtain every
Each set of fields that a field record region includes；And
Each set of fields that each field record region includes is arranged to corresponding sublist.
3. data processing method as described in claim 1, which is characterized in that the filling of the sublist includes:
According to the field attribute of the sublist, all fields of the sublist are divided, obtain that the sublist includes is each
The set of a field；
Judge whether the corresponding pending data of the data identification information is to be processed needed for the field value of field in sublist
Data；
If so, determining the fields match of the pending data and corresponding sublist, and it is filled into corresponding field area in sublist.
4. data processing method as described in claim 1, which is characterized in that described to carry out data to the filled sublist
Cleaning includes:
Consistency check is carried out to the data in the filled sublist, the consistency including the data in the single sublist
The consistency check of data between inspection and each sublist；
Invalid value inspection is carried out to the data in the filled sublist；And
Missing values inspection is carried out to the data in the filled sublist.
5. the data processing method as described in Claims 1-4 is any, which is characterized in that the update step includes:
The pending data filled in sublist is added to corresponding word in wide table by the corresponding relationship in field record region and sublist
Segment record region is updated；
Summarize field record region according in wide table, associated pending data is read from multiple sublists and carries out data remittance
Always, by the obtained data summarization be added in wide table described in summarize field record region and be updated.
6. a kind of electronic device, which is characterized in that the electronic device includes memory and processor, is stored in the memory
The data processor that can be run on the processor is realized as follows when the data processor is executed by the processor
Step:
Obtaining step: more pending datas in the first preset time are obtained from database, every pending data includes number
According to identification information；
Finding step: the mapping for each sublist and data identification information in multiple sublists for including according to pre-set wide table
Relationship finds out data identification information corresponding with each sublist in the pending data；
Filling step: according to the data identification information corresponding with each sublist found out from the pending data, by institute
It states the corresponding pending data of data identification information and is filled into corresponding sublist；
Verification step: data cleansing is carried out to the filled sublist, the sublist for completing data cleansing is saved in the number
According to library；
It updates step: receiving data summarization instruction, read this from the database and summarize the second specified preset time of instruction
Interior all sublists record the sublist of reading according to the corresponding relationship of different field record area and sublist in the wide table
Data be updated to the wide corresponding field record region of table.
7. electronic device as claimed in claim 6, which is characterized in that the foundation of the sublist includes:
The wide table is divided into a series of field record regions；
According to the field attribute in each field record region, all fields in each field record region are divided to obtain every
Each set of fields that a field record region includes；And
Each set of fields that each field record region includes is arranged to corresponding sublist.
8. electronic device as claimed in claim 6, which is characterized in that the filling of the sublist includes:
According to the field attribute of the sublist, all fields of the sublist are divided, obtain that the sublist includes is each
The set of a field；
Judge whether the corresponding pending data of the data identification information is to be processed needed for the field value of field in sublist
Data；
If so, determining the fields match of the pending data and corresponding sublist, and it is filled into corresponding field area in sublist.
9. electronic device as claimed in claim 6, which is characterized in that described to carry out data cleansing to the filled sublist
Include:
Consistency check is carried out to the data in the filled sublist, the consistency including the data in the single sublist
The consistency check of data between inspection and each sublist；
Invalid value inspection is carried out to the data in the filled sublist；And
Missing values inspection is carried out to the data in the filled sublist.
10. a kind of computer readable storage medium, which is characterized in that include data processing in the computer readable storage medium
Program, it can be achieved that data as described in any one of claim 1 to 5 when the data processor is executed by processor
The step of processing method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910525181.2A CN110399380A (en) | 2019-06-17 | 2019-06-17 | A kind of data processing method, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910525181.2A CN110399380A (en) | 2019-06-17 | 2019-06-17 | A kind of data processing method, electronic device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110399380A true CN110399380A (en) | 2019-11-01 |
Family
ID=68323215
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910525181.2A Pending CN110399380A (en) | 2019-06-17 | 2019-06-17 | A kind of data processing method, electronic device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110399380A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111008521A (en) * | 2019-12-06 | 2020-04-14 | 北京三快在线科技有限公司 | Method and device for generating wide table and computer storage medium |
| CN111274484A (en) * | 2020-01-20 | 2020-06-12 | 上海风秩科技有限公司 | Method and device for managing interactive data |
| CN111339063A (en) * | 2020-03-02 | 2020-06-26 | 中国平安人寿保险股份有限公司 | Intelligent management method and device for hypothesis data and computer readable storage medium |
| CN111552683A (en) * | 2020-04-23 | 2020-08-18 | 武汉澄川朗境环境科技有限公司 | Water affair data information management method and device based on big data |
| CN111737244A (en) * | 2020-06-22 | 2020-10-02 | 平安医疗健康管理股份有限公司 | Data quality inspection method, device, computer system and storage medium |
| CN112364021A (en) * | 2020-11-10 | 2021-02-12 | 中国平安人寿保险股份有限公司 | Service data processing method, device and storage medium |
| CN113157734A (en) * | 2021-04-20 | 2021-07-23 | 平安银行股份有限公司 | Data processing method, device and equipment based on search framework and storage medium |
| CN114049087A (en) * | 2021-11-15 | 2022-02-15 | 中国工商银行股份有限公司 | Processing method and device of transaction order data, storage medium and electronic equipment |
| CN117113950A (en) * | 2023-08-11 | 2023-11-24 | 广州标智未来科学技术有限公司 | High-throughput experimental data processing method and device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109558440A (en) * | 2018-10-18 | 2019-04-02 | 平安科技（深圳）有限公司 | Batch data processing method, device, computer equipment and storage medium |
| CN109656980A (en) * | 2018-12-27 | 2019-04-19 | Oppo（重庆）智能科技有限公司 | Data processing method, electronic equipment, device and readable storage medium storing program for executing |
-
2019
- 2019-06-17 CN CN201910525181.2A patent/CN110399380A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109558440A (en) * | 2018-10-18 | 2019-04-02 | 平安科技（深圳）有限公司 | Batch data processing method, device, computer equipment and storage medium |
| CN109656980A (en) * | 2018-12-27 | 2019-04-19 | Oppo（重庆）智能科技有限公司 | Data processing method, electronic equipment, device and readable storage medium storing program for executing |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111008521A (en) * | 2019-12-06 | 2020-04-14 | 北京三快在线科技有限公司 | Method and device for generating wide table and computer storage medium |
| CN111008521B (en) * | 2019-12-06 | 2023-04-28 | 北京三快在线科技有限公司 | Method, device and computer storage medium for generating wide table |
| CN111274484A (en) * | 2020-01-20 | 2020-06-12 | 上海风秩科技有限公司 | Method and device for managing interactive data |
| CN111339063A (en) * | 2020-03-02 | 2020-06-26 | 中国平安人寿保险股份有限公司 | Intelligent management method and device for hypothesis data and computer readable storage medium |
| CN111552683A (en) * | 2020-04-23 | 2020-08-18 | 武汉澄川朗境环境科技有限公司 | Water affair data information management method and device based on big data |
| CN111737244A (en) * | 2020-06-22 | 2020-10-02 | 平安医疗健康管理股份有限公司 | Data quality inspection method, device, computer system and storage medium |
| CN112364021A (en) * | 2020-11-10 | 2021-02-12 | 中国平安人寿保险股份有限公司 | Service data processing method, device and storage medium |
| CN112364021B (en) * | 2020-11-10 | 2023-10-13 | 中国平安人寿保险股份有限公司 | Service data processing method, device and storage medium |
| CN113157734A (en) * | 2021-04-20 | 2021-07-23 | 平安银行股份有限公司 | Data processing method, device and equipment based on search framework and storage medium |
| CN113157734B (en) * | 2021-04-20 | 2022-10-11 | 平安银行股份有限公司 | Data processing method, device and equipment based on search framework and storage medium |
| CN114049087A (en) * | 2021-11-15 | 2022-02-15 | 中国工商银行股份有限公司 | Processing method and device of transaction order data, storage medium and electronic equipment |
| CN117113950A (en) * | 2023-08-11 | 2023-11-24 | 广州标智未来科学技术有限公司 | High-throughput experimental data processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110399380A (en) | A kind of data processing method, electronic device and storage medium | |
| US20240169458A1 (en) | System and process for tokenization and management of liability | |
| CN109584082A (en) | Settlement of insurance claim method, electronic device and storage medium based on block chain | |
| CN107870980B (en) | Electronic device, billing data processing method and computer storage medium | |
| CN109815228A (en) | Creation method, device, computer equipment and the readable storage medium storing program for executing of database table | |
| EP2797013A1 (en) | Database update execution according to power management schemes | |
| CN110399333B (en) | Method, apparatus and computer program product for deleting snapshots | |
| CN108052279A (en) | A kind of method, apparatus, equipment and storage medium for promoting flash memory performance | |
| CN107402821A (en) | Access control method, device and the equipment of shared resource | |
| GB2500292A (en) | Managing a stack of identifiers of free blocks in a storage pool using a hash based linked list | |
| CN111681071A (en) | Sub-cost data generation system and method, storage medium, and electronic device | |
| CN110287695A (en) | A kind of management method of Java card and its temporary object | |
| CN110689333B (en) | Block chain automatic account checking method, device, system and storage medium | |
| CN116360695A (en) | Data storage method and device based on hybrid energy storage power supply | |
| CN110166530A (en) | Processing method, electronic device and computer equipment based on micro services return value | |
| CN111131393B (en) | User activity data statistical method, electronic device and storage medium | |
| CN110704488B (en) | Method for managing data and corresponding system, computer device and medium | |
| CN111429125B (en) | Account management method and device, storage medium and electronic equipment | |
| CN106874327A (en) | A kind of method of counting and device for business datum | |
| CN110427315A (en) | Push away excellent test device, method and storage medium | |
| US10050979B2 (en) | Executing a process based on validity information indicating one of validity and invalidity | |
| CN112347160B (en) | Work order management method, system and storage medium based on call center system | |
| CN111078714B (en) | Data processing method and device | |
| CN117668045A (en) | Blockchain data summarization storage method, device, equipment and storage medium | |
| CN116644045A (en) | Batch job task processing method and device, server and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |