CN108733662A - Method, apparatus, electronic equipment and the readable storage medium storing program for executing of comparison of data consistency - Google Patents
Method, apparatus, electronic equipment and the readable storage medium storing program for executing of comparison of data consistency Download PDFInfo
- Publication number
- CN108733662A CN108733662A CN201710239508.0A CN201710239508A CN108733662A CN 108733662 A CN108733662 A CN 108733662A CN 201710239508 A CN201710239508 A CN 201710239508A CN 108733662 A CN108733662 A CN 108733662A
- Authority
- CN
- China
- Prior art keywords
- data
- comparison
- consistency
- field
- key
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the present invention provides a kind of method, apparatus of comparison of data consistency, electronic equipment and readable storage medium storing program for executing, can improve the reusability, maintainability and scalability of comparison of data consistency tool.The comparison of data consistency method of the embodiment of the present invention includes：The initial data for being obtained from data source is parsed and is assigned to data object；The data object is mapped as the abstract data with unified format according to mapping ruler；Compare the consistency of the abstract data of the different data sources.
Description
Technical field
The present invention relates to field of computer technology more particularly to a kind of method, apparatus of comparison of data consistency, electronics to set
Standby and readable storage medium storing program for executing.
Background technology
For the scene of comparison of data consistency, from corresponding end downloading data, then mode fairly simple at present is
It is compared by comparison tool or alignment algorithm to realize.Such as shown in Fig. 1, for comparison of data consistency method in the prior art
Flow diagram.Respectively after data source A and data source B get data, data consistency is carried out by comparison tool/algorithm
It compares, and obtains consistency result.
Current general comparison of data consistency method, all customizes to obtain according to business scenario.Pass through data first
The interface that source provides realizes that corresponding logic obtains data source data, then reprocesses data, realize corresponding alignment algorithm with
Obtain the comparison result of consistency, last output-consistence result.
In realizing process of the present invention, inventor has found that at least there are the following problems in the prior art：
1. comparison method is obtained according to business customizing, reusability cannot be guaranteed, and need according to different business customizings
Different tools is to realize the comparison of data consistency；
2. the logic of method does not decouple, in Added Business, need to adapt to old service logic according to new business logic
Property modification, maintainable, poor expandability leads one and starts whole body, the adjustment of any point business that may all face the big of tool
Range is changed.
Invention content
In view of this, the embodiment of the present invention provides a kind of method, apparatus of comparison of data consistency, electronic equipment and readable
Storage medium can improve the reusability, maintainability and scalability of comparison of data consistency tool.
To achieve the above object, one side according to the ... of the embodiment of the present invention provides a kind of comparison of data consistency
Method.
The comparison of data consistency method of the embodiment of the present invention includes：The initial data for being obtained from data source is parsed and assigned
It is worth to data object；The data object is mapped as the abstract data with unified format according to mapping ruler；It compares different
The consistency of the abstract data of the data source.
Optionally, the initial data for being obtained from data source is parsed and is assigned to data object, including：Each institute is defined respectively
State the data processing rule of data source；According to the data processing rule, the initial data for being obtained from the data source is parsed
And it is assigned to the data object.
Optionally, the data object is mapped as by the abstract data with unified format according to mapping ruler, including：It obtains
Take field to be compared, using the field name of the field to be compared as key, using content corresponding with the field name as value, build
Vertical key-value pair；The data object is parsed to obtain the whole fields for including in the data object；From whole fields
Selection field identical with the field name of field to be compared is simultaneously stored in the key-value pair.
Optionally, the consistency of the abstract data of the different data sources is compared, including：Compare different data sources
In the key-value pair, the consistency of the value corresponding to same keys.
Optionally, the method further includes：Report data is generated according to comparison result.
To achieve the above object, other side according to the ... of the embodiment of the present invention provides a kind of comparison of data consistency
Device.
The device of the comparison of data consistency of the embodiment of the present invention, including：Data processing module, for data will to be obtained from
The initial data in source parses and is assigned to data object；Data mapping module, for according to mapping ruler by the data object
It is mapped as the abstract data with unified format；Data comparing module, the abstract number for comparing the different data sources
According to consistency.
Optionally, the data processing module is additionally operable to：The data processing rule of each data source is defined respectively；According to
The initial data for being obtained from the data source is parsed and is assigned to the data object by the data processing rule.
Optionally, the data mapping module is additionally operable to：Field to be compared is obtained, with the field name of the field to be compared
As key, using content corresponding with the field name as value, establish key-value pair；The data object is parsed to obtain the number
According to the whole fields for including in object；The selection field identical with the field name of field to be compared from whole fields
And it is stored in the key-value pair.
Optionally, the uniformity comparison module is additionally operable to：In the key-value pair for comparing different data sources, same keys institute
The consistency of corresponding value.
Optionally, described device further includes：Report generation module, for generating report data according to comparison result.
To achieve the above object, another aspect according to the ... of the embodiment of the present invention provides a kind of comparison of data consistency
Electronic equipment, including：
One or more processors；
Storage device, for storing one or more programs,
When one or more of programs are executed by one or more of processors so that one or more of processing
Device is at least realized：
The initial data for being obtained from data source is parsed and is assigned to data object；According to mapping ruler by the data pair
As being mapped as the abstract data with unified format；Compare the consistency of the abstract data of the different data sources.
To achieve the above object, another aspect according to the ... of the embodiment of the present invention provides a kind of computer-readable medium,
On be stored with computer program, at least realized when described program is executed by processor：
The initial data for being obtained from data source is parsed and is assigned to data object；According to mapping ruler by the data pair
As being mapped as the abstract data with unified format；Compare the consistency of the abstract data of the different data sources.
One embodiment in foregoing invention has the following advantages that or advantageous effect：Because using the original of different-format
Beginning data are mapped as the technological means being compared after the abstract data of same format, solve the prior art need by with number
The problem of according to the corresponding logic in source and being processed, realizes the decoupling of data mart modeling and comparing, and having reached raising can answer
With the technique effect of property；Because using the technological means that the carrier using data object as initial data transmits initial data,
The decoupling that initial data is obtained and processed is realized, the business of stale source can be patrolled by solving when the prior art increases data source newly
The problem of impacting is collected, has reached and has improved maintainable, scalability technique effect.
Further effect possessed by above-mentioned non-usual optional mode adds hereinafter in conjunction with specific implementation mode
With explanation.
Description of the drawings
Attached drawing does not constitute inappropriate limitation of the present invention for more fully understanding the present invention.Wherein：
Fig. 1 is the flow diagram of comparison of data consistency method in the prior art；
Fig. 2 is the schematic diagram of the key step of the method for comparison of data consistency according to the ... of the embodiment of the present invention；
Fig. 3 is the schematic diagram of the main modular of the device of comparison of data consistency according to the ... of the embodiment of the present invention；
Fig. 4 is the schematic diagram of another implementation method of the device of comparison of data consistency according to the ... of the embodiment of the present invention；
Fig. 5 is adapted for the structural representation of the computer system for the terminal device or server of realizing the embodiment of the present invention
Figure.
Specific implementation mode
It explains to the exemplary embodiment of the present invention below in conjunction with attached drawing, including the various of the embodiment of the present invention
Details should think them only exemplary to help understanding.Therefore, those of ordinary skill in the art should recognize
It arrives, various changes and modifications can be made to the embodiments described herein, without departing from scope and spirit of the present invention.Together
The description to known function and structure is omitted for clarity and conciseness in sample in following description.
Existing comparison of data consistency technology realizes that corresponding logic obtains data source by the interface that data source provides
Then data reprocess data, realize corresponding alignment algorithm to obtain the comparison result of consistency, last output-consistence knot
Fruit.Due to needing to customize to obtain according to business scenario, logic does not decouple, and source of new data logic can cause shadow to stale source logic
It rings, reusability, maintainability and scalability are poor.The present invention considers the defect of the prior art, by data acquisition, number
It is decoupled respectively according to processing and data comparison process, when new data source is added, as long as the processing logic of newly-increased source of new data, and
The processing logic of stale source need not be adaptively adjusted, to reach more good reusability, maintainability and can
Autgmentability, while also greatly simplifying the uniformity comparison logic between 2 data above sources.
Fig. 2 is the schematic diagram of the key step of the method for comparison of data consistency according to the ... of the embodiment of the present invention；
As shown in Fig. 2, the embodiment of the present invention provides a kind of method of comparison of data consistency, including：
The initial data for being obtained from data source is parsed and is assigned to data object by S20.Need which obtains from data source
Initial data is predefined according to business；For different data sources, due to its system architecture difference, number
According to preservation form it is different, therefore the form of the initial data got also differs；By parsing initial data, release original
The service logic of data is translated into the data for being more prone to compare, and is assigned to data object, is used as by data object
The carrier of initial data, to realize logic decoupling.
The data object is mapped as the abstract data with unified format by S21 according to mapping ruler.From different data
The form for the initial data that source is got simultaneously differs, and can not directly be compared；Even across parsing and be assigned to data pair
It is entirely data to be compared not also to be as the data for later, including in data object, and calculation amount will necessarily be increased by directly comparing.
Therefore data object is mapped as abstract data by this step；Abstract data refer to according to data type as defined in unified requirement,
By the way that data object is mapped as abstract data, the format when data of multiterminal data source enter memory is unified, has been convenient for data
Preservation and follow-up compare.When increasing new data source, as long as accordingly increasing the data object of the data source to unified pumping
The mapping method of image data, so that it may compare patrolling for step without changing to complete uniformity comparison according to existing comparison step
Volume.In addition, when mapping data object to obtain abstract data, it is contemplated that simplify follow-up comparison step, it can also be further right
Abstract data such as is marked, classifies, sorting at the operations, to further increase the comparable of the obtained abstract data of different data sources
Property, reduce processing step when comparing.
S22 compares the consistency of the abstract data of the different data sources.
From the above it can be seen that the method for comparison of data consistency provided in this embodiment, it will not because using
Initial data with format parses and is assigned to data object, after data object is then mapped as the abstract data of same format
The technological means being compared solves the problems, such as that the prior art is needed through logic corresponding with data source and is processed,
The decoupling for realizing data mart modeling and comparing has reached the technique effect for improving reusability.
In some optional embodiments, S20 obtains initial data from data source, including：
The data processing rule of each data source is defined respectively；According to the data processing rule, will be obtained from described
The initial data of data source parses and is assigned to the data object.
Realization method of the data object in different language is different, can be a variable, can also be that a class is realized
Object.Connected each other by preset data processing rule between data object and the initial data of the data source, from
After data source gets initial data, initial data is parsed according to data processing rule, and by the result after parsing
It is assigned to data object.By defining data processing rule, contact of the different data sources to data object can be established, will include
The initial data of different business logic resolves to the data object not comprising service logic, to realize data acquisition and data
The decoupling of comparison.In newly-increased data source, it is only necessary to increase the initial data of the data source newly at the data of specified data object
Reason rule is impacted without the data acquisition to stale source, comparison logic.
From the above it can be seen that the present embodiment is because the carrier used using data object as initial data transmits
The technological means of initial data, and the data processing rule from initial data to data object is formulated respectively to each data source,
The decoupling for realizing initial data acquisition and comparing, solving can be to the industry of stale source when the prior art increases data source newly
The problem of business logic impacts has reached and has improved maintainable, scalability technique effect.
In some optional embodiments, the data object is mapped as with unified format by S21 according to mapping ruler
Abstract data, including：
Field to be compared is obtained, using the field name of the field to be compared as key, with corresponding with the field name interior
Hold as value, establishes key-value pair.Field is a part for the data that data object is included and the master of comparison of data consistency
Body.Field to be compared refers to the field for needing to be compared, and is predefined as needed；Field to be compared usually has multiple, is
Convenient for comparing, each field to be compared is correspondingly provided with field name, and the content of field is the data that the field is included.
Key-value pair includes key (Key) and value (Value) corresponding with key, is a kind of form (such as Map key-value pairs etc.) of data storage,
By searching for key, you can rapidly obtain the value opposite with key.
The data object is parsed to obtain the whole fields for including in the data object；It is selected from whole fields
It selects field identical with the field name of field to be compared and is stored in the key-value pair.The Field Count for including in data object
May be many according to amount, but the only above-mentioned field to be compared for participating in comparing is actually needed；For raising arithmetic speed etc.
Consider, primary screening is carried out to initial data, only chooses field identical with field to be compared, and these fields are stored in front
In the key-value pair set of foundation, to realize simplifying for data.
It should be noted that certain fields in field to be compared may be not present in the field that data object is included.
Such as we realize the comparison of two end (data terminal A, data terminal B) data of a business：Data terminal A there are service fields a, b,
C, d, e, f, there are service fields a, b, d, e, f, g, h by data terminal B.It is a, b, c, d, e that we, which need the field compared, so handle
The service fields of data terminal A are mapped to a, b, c, d, the data object of e fields, while also the service fields of data terminal B are mapped
At a, b, c, d, e.It can see data terminal B and field c be not present, therefore parse the initial data of data terminal B and be stored in key assignments
When to set, corresponding c fields can leave a blank, or assign an identifiable value, can easily know when comparing
Difference is not gone out.In addition, the process of mapping includes simple extraction section field, while can also be that some industry are carried out to specific word
It is extracted again after the operation for logic of being engaged in.
In some optional embodiments, S22 compares the consistency of the abstract data of the different data sources, packet
It includes：
In the key-value pair for comparing different data sources, the consistency of the value corresponding to same keys.It should be noted that
Compare consistency when, can directly compare, be compared again after can also being ranked up first, relative to directly progress both ends/
Multiterminal directly come to comparing, and the mode compared again after sequence can reduce time complexity.The time complexity of sort algorithm can
To accomplish Nlog2N, the time complexity of the comparison after sequence is linear N, the comparison for two end datas, and overall time is multiple
Miscellaneous degree is N+Nlog2N, if without directly comparison of sorting, time complexity is N2.The case where for multiterminal, sequence are calculated
The advantage of the time complexity of method can bigger.
In some optional embodiments, the method further includes：
Report data is generated according to comparison result.It, can also be into one for the ease of inquiry etc. after generating report data
Report data is uploaded to Cloud Server by step, or further according to the corresponding report postal of the generations such as the permission of different accounts and demand
Part is simultaneously sent.
In other alternative-embodiments, the method further includes：
According to preset time interval, timing compares the consistency of the specified data in preset data source.
From the above it can be seen that the initial data of different-format is mapped as phase by the embodiment of the present invention because using
With the technological means being compared after the abstract data of format, solves the prior art and need through logic corresponding with data source
And the problem of being processed, the decoupling of data mart modeling and comparing is realized, the technique effect for improving reusability has been reached；
Because using the technological means that the carrier using data object as initial data transmits initial data, realizes initial data and obtain
The decoupling for taking and processing, solving when the prior art increases data source newly can ask what the service logic of stale source impacted
Topic has reached and has improved maintainable, scalability technique effect.
Fig. 3 is the schematic diagram of the main modular of the device of comparison of data consistency according to the ... of the embodiment of the present invention.
As shown in figure 3, the embodiment of the present invention provides a kind of device 30 of comparison of data consistency, including data processing module
301, data mapping module 302 and data comparing module 303, wherein：Data processing module 301, for data source will to be obtained from
Initial data parse and be assigned to data object；Data mapping module 302, for according to mapping ruler by the data object
It is mapped as the abstract data with unified format；Data comparing module 303, for comparing the described abstract of the different data sources
The consistency of data.
In some optional embodiments, the data processing module 301 is additionally operable to：Each data source is defined respectively
Data processing rule；According to the data processing rule, the initial data for being obtained from the data source is parsed and is assigned to institute
State data object.
In some optional embodiments, the data mapping module 302 is additionally operable to：Field to be compared is obtained, with described
The field name of field to be compared as key, using content corresponding with the field name as value, establish key-value pair；Parse the number
According to object to obtain the whole fields for including in the data object；Selection and the field to be compared from whole fields
The identical field of field name and be stored in the key-value pair.
In some optional embodiments, the uniformity comparison module 303 is additionally operable to：Compare the described of different data sources
In key-value pair, the consistency of the value corresponding to same keys.
In some optional embodiments, described device further includes report generation module 304, for being given birth to according to comparison result
At report data.
Fig. 4 is the schematic diagram of another implementation method of the device of comparison of data consistency according to the ... of the embodiment of the present invention.
As shown in figure 4, establishing device of the uniformity comparison engine as uniformity comparison.Uniformity comparison engine 40 wraps
Contain data source modules 401, parsing mapping block 402, uniformity comparison module 403, Reports module 404 and other expansion modules
405；The function of each module is as follows：Data source modules 401：Definition obtains original number from uniformity comparison engine 40 to each data source
According to rule, at newly-increased comparing task/data source, may customize 401 pieces new of data source mould be supported, guarantee can expand
Malleability；Parse mapping block 402：The initial data of data source is handled to the mapping of abstract data, when being compared, to multiterminal
Abstract data is compared, and simplifies uniformity comparison algorithm；Uniformity comparison module 403：Define uniformity comparison rule and
Algorithm, the abstract data formed after being processed to initial data is compared, and comparison result is returned to Reports module 404, into
Row report output can design the realizations of a variety of comparison algorithms in uniformity comparison module；Abstract data refers to unified format
Data can be converted into abstract data after initial data processing, are compared in next module；Reports module 404：According to one
The comparison result that cause property comparing module 403 exports, generates the result form needed for different business scene；Other expansion modules
405：Uniformity comparison engine can be extended, add different modules and be supported.
The function of uniformity comparison engine 40 is further illustrated below by a specific embodiment.
In a material handles scene, needs to safeguard local material data, be related to local material data and exterior flat
Number of units according to both ends comparison of data consistency.In this scenario, each module in the uniformity comparison engine 40 is realized as follows
Function：
Data source modules 401：Local data source modules configure corresponding Mysql data sources, pass through DAO layers of acquisition material
The data source modules of data, platform obtain corresponding material csv file by calling platform API, are assigned to after parsing corresponding
Data object supports the extension of multiple data sources.
Parse mapping block 402：The data object obtained by data source modules 401, in parsing mapping block 402, root
Demand is compared according to business, is mapped to corresponding Map key-value pairs, such as local data generates KeywordObject, platform data
KeywordObject is generated, being passed to uniformity comparison module after unified field is compared.Unified field means that guarantee is reflected
Key fields is consistent in the key-value pair penetrated, and only needs the value values for extracting corresponding key to be compared in comparison process, no
It needs to pay close attention to service logic again.
Uniformity comparison module 403：403 define corresponding alignment algorithm in uniformity comparison module, are realized in business
File internal sort, data object is compared one by one again after sequence mode.
Reports module 404：Reports module exports after being formatted to the result after corresponding uniformity comparison, such as by one
Cause property result is formatted as csv file report as a result, shows the consistency of every material data as a result, and by result form
The high in the clouds OSS is uploaded to, corresponding report mail is finally generated according to account dimension.
Other expansion modules 405：In the case where each module decouples, can by way of adding new module to business into
Row is supported, for example adds timer module, and timing executes uniformity comparison task, can also similarly add other moulds as desired
Block carries out business support.
Comparison of data consistency engine after optimizing, data source modules obtain initial data from multiterminal, by mapping
Processing module simplifies the abstract data for needing the field compared and being processed into unified format.It is then transferred to uniformity comparison mould
Block realizes the comparison of real consistency.The report data needed for different business scene is finally generated in Reports module.
From the above it can be seen that the initial data of different-format is mapped as phase by the embodiment of the present invention because using
With the technological means being compared after the abstract data of format, solves the prior art and need through logic corresponding with data source
And the problem of being processed, the decoupling of data mart modeling and comparing is realized, the technique effect for improving reusability has been reached；
Because using the technological means that the carrier using data object as initial data transmits initial data, realizes initial data and obtain
The decoupling for taking and processing, solving when the prior art increases data source newly can ask what the service logic of stale source impacted
Topic has reached and has improved maintainable, scalability technique effect.
According to an embodiment of the invention, the present invention also provides a kind of electronic equipment and a kind of readable storage medium storing program for executing.
Fig. 5 is adapted for the structural representation of the computer system for the terminal device or server of realizing the embodiment of the present invention
Figure.
Below with reference to Fig. 5, it illustrates the computer systems 500 suitable for the terminal device for realizing the embodiment of the present invention
Structural schematic diagram.Terminal device shown in Fig. 5 is only an example, to the function of the embodiment of the present invention and should not use model
Shroud carrys out any restrictions.
As shown in figure 5, computer system 500 includes central processing unit (CPU) 501, it can be read-only according to being stored in
Program in memory (ROM) 502 or be loaded into the program in random access storage device (RAM) 503 from storage section 508 and
Execute various actions appropriate and processing.In RAM 503, also it is stored with system 500 and operates required various programs and data.
CPU 501, ROM 502 and RAM 503 are connected with each other by bus 504.Input/output (I/O) interface 505 is also connected to always
Line 504.
It is connected to I/O interfaces 505 with lower component：Importation 506 including keyboard, mouse etc.；It is penetrated including such as cathode
The output par, c 507 of spool (CRT), liquid crystal display (LCD) etc. and loud speaker etc.；Storage section 508 including hard disk etc.；
And the communications portion 509 of the network interface card including LAN card, modem etc..Communications portion 509 via such as because
The network of spy's net executes communication process.Driver 510 is also according to needing to be connected to I/O interfaces 505.Detachable media 511, such as
Disk, CD, magneto-optic disk, semiconductor memory etc. are mounted on driver 510, as needed in order to be read from thereon
Computer program be mounted into storage section 508 as needed.
Particularly, according to an embodiment of the invention, the process of the schematic diagram description of key step above may be implemented as
Computer software programs.For example, the embodiment of the present invention includes a kind of computer program product comprising being carried on computer can
The computer program on medium is read, which includes the program for executing method shown in the schematic diagram of key step
Code.In such embodiments, which can be downloaded and installed by communications portion 509 from network, and/
Or it is mounted from detachable media 511.When the computer program is executed by central processing unit (CPU) 501, the present invention is executed
System in the above-mentioned function that limits.
It should be noted that computer-readable medium shown in the present invention can be computer-readable signal media or meter
Calculation machine readable storage medium storing program for executing either the two arbitrarily combines.Computer readable storage medium for example can be --- but not
Be limited to --- electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor system, device or device, or arbitrary above combination.Meter
The more specific example of calculation machine readable storage medium storing program for executing can include but is not limited to：Electrical connection with one or more conducting wires, just
It takes formula computer disk, hard disk, random access storage device (RAM), read-only memory (ROM), erasable type and may be programmed read-only storage
Device (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), light storage device, magnetic memory device,
Or above-mentioned any appropriate combination.In the present invention, can be any include computer readable storage medium or storage journey
The tangible medium of sequence, the program can be commanded the either device use or in connection of execution system, device.And at this
In invention, computer-readable signal media may include in a base band or as the data-signal that a carrier wave part is propagated,
Wherein carry computer-readable program code.Diversified forms may be used in the data-signal of this propagation, including but unlimited
In electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be that computer can
Any computer-readable medium other than storage medium is read, which can send, propagates or transmit and be used for
By instruction execution system, device either device use or program in connection.Include on computer-readable medium
Program code can transmit with any suitable medium, including but not limited to：Wirelessly, electric wire, optical cable, RF etc. or above-mentioned
Any appropriate combination.
Flow chart in attached drawing and block diagram, it is illustrated that according to the system of various embodiments of the invention, method and computer journey
The architecture, function and operation in the cards of sequence product.In this regard, each box in flowchart or block diagram can generation
A part for a part for one module, program segment, or code of table, above-mentioned module, program segment, or code includes one or more
Executable instruction for implementing the specified logical function.It should also be noted that in some implementations as replacements, institute in box
The function of mark can also occur in a different order than that indicated in the drawings.For example, two boxes succeedingly indicated are practical
On can be basically executed in parallel, they can also be executed in the opposite order sometimes, this is depended on the functions involved.Also it wants
It is noted that the combination of each box in block diagram or flow chart and the box in block diagram or flow chart, can use and execute rule
The dedicated hardware based systems of fixed functions or operations is realized, or can use the group of specialized hardware and computer instruction
It closes to realize.
Being described in module involved in the embodiment of the present invention can be realized by way of software, can also be by hard
The mode of part is realized.Described module can also be arranged in the processor, for example, can be described as：A kind of processor packet
Include data processing module, data mapping module, data comparing module.Wherein, the title of these units is not under certain conditions
The restriction to the unit itself is constituted, " being used for will be described according to mapping ruler for example, data mapping module is also described as
Data object is mapped as the module of the abstract data with unified format ".
As on the other hand, the present invention also provides a kind of computer-readable medium, which can be
Included in equipment described in above-described embodiment；Can also be individualism, and without be incorporated the equipment in.Above-mentioned calculating
Machine readable medium carries one or more program, when said one or multiple programs are executed by the equipment, makes
Obtaining the equipment includes：
The initial data for being obtained from data source is parsed and is assigned to data object；According to mapping ruler by the data pair
As being mapped as the abstract data with unified format；Compare the consistency of the abstract data of the different data sources.
The initial data of different-format is mapped as phase apposition by technical solution according to the ... of the embodiment of the present invention because using
The technological means being compared after the abstract data of formula solves prior art needs and is gone forward side by side by logic corresponding with data source
The problem of row processing, the decoupling of data mart modeling and comparing is realized, has reached the technique effect for improving reusability；Because
Use the technological means that carrier using data object as initial data transmits initial data, realize initial data obtain and
The decoupling of processing is solved the problems, such as to impact the service logic of stale source when the prior art increases data source newly, be reached
It has arrived and has improved maintainable, scalability technique effect.
Above-mentioned specific implementation mode, does not constitute limiting the scope of the invention.Those skilled in the art should be bright
It is white, design requirement and other factors are depended on, various modifications, combination, sub-portfolio and replacement can occur.It is any
Modifications, equivalent substitutions and improvements made by within the spirit and principles in the present invention etc., should be included in the scope of the present invention
Within.
Claims (12)
1. a kind of method of comparison of data consistency, which is characterized in that including：
The initial data for being obtained from data source is parsed and is assigned to data object；
The data object is mapped as the abstract data with unified format according to mapping ruler；
Compare the consistency of the abstract data of the different data sources.
2. according to the method described in claim 1, it is characterized in that, the initial data for being obtained from data source being parsed and being assigned to
Data object, including：
The data processing rule of each data source is defined respectively；
According to the data processing rule, the initial data for being obtained from the data source is parsed and is assigned to the data pair
As.
3. according to the method described in claim 1, it is characterized in that, the data object is mapped as having according to mapping ruler
The abstract data of unified format, including：
Field to be compared is obtained, is made using the field name of the field to be compared as key, with content corresponding with the field name
For value, key-value pair is established；
The data object is parsed to obtain the whole fields for including in the data object；
Field identical with the field name of field to be compared is selected from whole fields and is stored in the key-value pair.
4. according to the method described in claim 3, it is characterized in that, comparing the one of the abstract data of the different data sources
Cause property, including：
In the key-value pair for comparing different data sources, the consistency of the value corresponding to same keys.
5. according to the method described in claim 1, it is characterized in that, the method further includes：
Report data is generated according to comparison result.
6. a kind of device of comparison of data consistency, which is characterized in that including：
Data processing module, for being parsed the initial data for being obtained from data source and being assigned to data object；
Data mapping module, for the data object to be mapped as the abstract data with unified format according to mapping ruler；
Data comparing module, the consistency of the abstract data for comparing the different data sources.
7. device according to claim 6, which is characterized in that the data processing module is additionally operable to：Each institute is defined respectively
State the data processing rule of data source；According to the data processing rule, the initial data for being obtained from the data source is parsed
And it is assigned to the data object.
8. device according to claim 6, which is characterized in that the data mapping module is additionally operable to：Obtain word to be compared
Section, using the field name of the field to be compared as key, using content corresponding with the field name as be worth, establish key-value pair；
The data object is parsed to obtain the whole fields for including in the data object；From whole fields selection with it is described
The identical field of field name of field to be compared is simultaneously stored in the key-value pair.
9. device according to claim 8, which is characterized in that the uniformity comparison module is additionally operable to：Compare different numbers
According in the key-value pair in source, the consistency of the value corresponding to same keys.
10. device according to claim 6, which is characterized in that described device further includes：
Report generation module, for generating report data according to comparison result.
11. a kind of electronic equipment of comparison of data consistency, which is characterized in that including：
One or more processors；
Storage device, for storing one or more programs,
When one or more of programs are executed by one or more of processors so that one or more of processors are real
The now method as described in any in claim 1-5.
12. a kind of computer-readable medium, is stored thereon with computer program, which is characterized in that described program is held by processor
The method as described in any in claim 1-5 is realized when row.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710239508.0A CN108733662A (en) | 2017-04-13 | 2017-04-13 | Method, apparatus, electronic equipment and the readable storage medium storing program for executing of comparison of data consistency |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710239508.0A CN108733662A (en) | 2017-04-13 | 2017-04-13 | Method, apparatus, electronic equipment and the readable storage medium storing program for executing of comparison of data consistency |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108733662A true CN108733662A (en) | 2018-11-02 |
Family
ID=63923752
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710239508.0A Pending CN108733662A (en) | 2017-04-13 | 2017-04-13 | Method, apparatus, electronic equipment and the readable storage medium storing program for executing of comparison of data consistency |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108733662A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109616212A (en) * | 2018-11-09 | 2019-04-12 | 金色熊猫有限公司 | Disease data processing method, device, electronic equipment and readable medium |
| CN110297842A (en) * | 2019-06-25 | 2019-10-01 | 武汉联影医疗科技有限公司 | A kind of data comparison method, device, terminal and storage medium |
| CN110704325A (en) * | 2019-10-09 | 2020-01-17 | 京东数字科技控股有限公司 | Data processing method and device, computer storage medium and electronic equipment |
| CN111324648A (en) * | 2020-01-21 | 2020-06-23 | 北京工业大数据创新中心有限公司 | Method and device for processing time sequence data |
| CN111563088A (en) * | 2020-04-20 | 2020-08-21 | 成都库珀区块链科技有限公司 | Data consistency detection method and device |
| CN111831534A (en) * | 2019-04-15 | 2020-10-27 | 西安京迅递供应链科技有限公司 | Method and device for verifying accuracy of datagram table |
| CN113407551A (en) * | 2020-03-17 | 2021-09-17 | 北京同邦卓益科技有限公司 | Data consistency determining method, device, equipment and storage medium |
| CN113742322A (en) * | 2020-11-17 | 2021-12-03 | 北京沃东天骏信息技术有限公司 | Data quality detection method and device |
| CN117435509A (en) * | 2023-12-20 | 2024-01-23 | 深圳市智慧城市科技发展集团有限公司 | Dynamic comparison method, dynamic comparison device and storage medium for interface data |
| US11941023B2 (en) | 2019-06-26 | 2024-03-26 | Zte Corporation | System and method for implementing incremental data comparison |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104036029A (en) * | 2014-06-27 | 2014-09-10 | 深圳市腾讯计算机***有限公司 | Big data consistency comparison method and system |
| CN104200402A (en) * | 2014-09-11 | 2014-12-10 | 国家电网公司 | Publishing method and system of source data of multiple data sources in power grid |
| CN105069159A (en) * | 2015-08-26 | 2015-11-18 | 浪潮通用软件有限公司 | Extensible data consistency check tool |
| US20160292206A1 (en) * | 2015-03-31 | 2016-10-06 | Oracle International Corporation | Validating coherency between multiple data sets between database transfers |
| CN106202111A (en) * | 2015-05-06 | 2016-12-07 | 阿里巴巴集团控股有限公司 | The method of calibration of database data and device |
-
2017
- 2017-04-13 CN CN201710239508.0A patent/CN108733662A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104036029A (en) * | 2014-06-27 | 2014-09-10 | 深圳市腾讯计算机***有限公司 | Big data consistency comparison method and system |
| CN104200402A (en) * | 2014-09-11 | 2014-12-10 | 国家电网公司 | Publishing method and system of source data of multiple data sources in power grid |
| US20160292206A1 (en) * | 2015-03-31 | 2016-10-06 | Oracle International Corporation | Validating coherency between multiple data sets between database transfers |
| CN106202111A (en) * | 2015-05-06 | 2016-12-07 | 阿里巴巴集团控股有限公司 | The method of calibration of database data and device |
| CN105069159A (en) * | 2015-08-26 | 2015-11-18 | 浪潮通用软件有限公司 | Extensible data consistency check tool |
Non-Patent Citations (1)
| Title |
|---|
| MIKE FROST: "Solving data access problems in SOA environments", 《WWW.DEVELOPERFUSION/ARTICLE/84928/SOLVING-DATA-ACCESS-PROBLEMS-IN-SOA-ENVIRONMENTS/》 * |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109616212A (en) * | 2018-11-09 | 2019-04-12 | 金色熊猫有限公司 | Disease data processing method, device, electronic equipment and readable medium |
| CN111831534A (en) * | 2019-04-15 | 2020-10-27 | 西安京迅递供应链科技有限公司 | Method and device for verifying accuracy of datagram table |
| CN110297842A (en) * | 2019-06-25 | 2019-10-01 | 武汉联影医疗科技有限公司 | A kind of data comparison method, device, terminal and storage medium |
| US11941023B2 (en) | 2019-06-26 | 2024-03-26 | Zte Corporation | System and method for implementing incremental data comparison |
| CN110704325A (en) * | 2019-10-09 | 2020-01-17 | 京东数字科技控股有限公司 | Data processing method and device, computer storage medium and electronic equipment |
| CN110704325B (en) * | 2019-10-09 | 2021-07-30 | 京东数字科技控股有限公司 | Data processing method and device, computer storage medium and electronic equipment |
| CN111324648A (en) * | 2020-01-21 | 2020-06-23 | 北京工业大数据创新中心有限公司 | Method and device for processing time sequence data |
| CN113407551A (en) * | 2020-03-17 | 2021-09-17 | 北京同邦卓益科技有限公司 | Data consistency determining method, device, equipment and storage medium |
| CN111563088A (en) * | 2020-04-20 | 2020-08-21 | 成都库珀区块链科技有限公司 | Data consistency detection method and device |
| CN113742322A (en) * | 2020-11-17 | 2021-12-03 | 北京沃东天骏信息技术有限公司 | Data quality detection method and device |
| CN117435509A (en) * | 2023-12-20 | 2024-01-23 | 深圳市智慧城市科技发展集团有限公司 | Dynamic comparison method, dynamic comparison device and storage medium for interface data |
| CN117435509B (en) * | 2023-12-20 | 2024-04-02 | 深圳市智慧城市科技发展集团有限公司 | Dynamic comparison method, dynamic comparison device and storage medium for interface data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108733662A (en) | Method, apparatus, electronic equipment and the readable storage medium storing program for executing of comparison of data consistency | |
| US20210357814A1 (en) | Method for distributed training model, relevant apparatus, and computer readable storage medium | |
| CN108683539A (en) | Management method, device, medium and the electronic equipment of block chain network | |
| CN109299458A (en) | Entity recognition method, device, equipment and storage medium | |
| CN110472207A (en) | List generation method and device | |
| CN109189379A (en) | code generating method and device | |
| CN108629823A (en) | The generation method and device of multi-view image | |
| CN108830329A (en) | Image processing method and device | |
| CN108510081A (en) | machine learning method and platform | |
| CN109981322A (en) | The method and apparatus of cloud resource management based on label | |
| CN107341235A (en) | Report form inquiring method, device, medium and electronic equipment | |
| CN109815107A (en) | The method and apparatus of automatic test | |
| CN108776692A (en) | Method and apparatus for handling information | |
| CN109993179A (en) | The method and apparatus that a kind of pair of data are clustered | |
| CN106202438A (en) | The method and system of storage associated data | |
| CN109903105A (en) | A kind of method and apparatus for improving end article attribute | |
| CN108984496A (en) | The method and apparatus for generating report | |
| CN109284342A (en) | Method and apparatus for output information | |
| CN108920618A (en) | A kind of store method, device, medium and the electronic equipment of object reference relationship | |
| CN110309142A (en) | The method and apparatus of regulation management | |
| CN110109912A (en) | A kind of identifier generation method and device | |
| CN107798556A (en) | For updating method, equipment and the storage medium of situation record | |
| CN108810130A (en) | A kind of method and apparatus of distribution request planning | |
| CN110019531A (en) | A kind of method and apparatus obtaining analogical object set | |
| CN115391343A (en) | Bill data processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181102 |