CN110162544A - Heterogeneous data source data capture method and device - Google Patents
Heterogeneous data source data capture method and device Download PDFInfo
- Publication number
- CN110162544A CN110162544A CN201910465165.9A CN201910465165A CN110162544A CN 110162544 A CN110162544 A CN 110162544A CN 201910465165 A CN201910465165 A CN 201910465165A CN 110162544 A CN110162544 A CN 110162544A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- tables
- integration
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24558—Binary matching operations
- G06F16/2456—Join operations
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of heterogeneous data source data capture method and device, method includes: the integration across database data inquiry request for receiving user and sending；According to integration across database data inquiry request, the correlation inquiry information of each tables of data of integration across database is determined；Correlation inquiry information is parsed, the data table information in related each database is determined, and generate corresponding query statement for related tables of data, to obtain the query result of corresponding tables of data；According to correlation inquiry information, the query result for being associated with each tables of data carries out secondary inquiry, obtains the corresponding query result of integration across database data inquiry request.According to the present invention it is possible to which user is helped to realize that the data query of integration across database obtains query result so that user is not necessarily to pay close attention to the query language of disparate databases, Stored Data Type not parity problem.
Description
Technical field
The present invention relates to database fields, and in particular to a kind of heterogeneous data source data capture method and device.
Background technique
With the fast development of business, portfolio increases.Business can generate a large amount of data when executing, so that data inventory
The data volume of storage is more and more huger.Sometimes, the data that different business scene generates can be stored according to the difference of business scenario
In different databases.Make the storage of business datum can be in heterogeneous data source in this way.Heterogeneous data source such as Explorer,
The different types of database such as Mysql, ODPS, HBase.For the user for needing to inquire data, inquiry may be related to
Data are inquired from multiple and different heterogeneous data sources, disparate databases, which exist, in heterogeneous data source inquires used language not
With the data type stored in, database not parity problem, cause user query difficult.
Summary of the invention
In view of the above problems, it proposes on the present invention overcomes the above problem or at least be partially solved in order to provide one kind
State the heterogeneous data source data capture method and device of problem.
According to an aspect of the invention, there is provided a kind of heterogeneous data source data capture method, wrapped in heterogeneous data source
The database of multiple and different types is included, method includes:
Receive the integration across database data inquiry request that user sends；
According to integration across database data inquiry request, the correlation inquiry information of each tables of data of integration across database is determined；
Correlation inquiry information is parsed, determines the data table information in related each database, and be involved
Tables of data generate corresponding query statement, to obtain the query result of corresponding tables of data；According to correlation inquiry information, association
The query result of each tables of data carries out secondary inquiry, obtains the corresponding query result of integration across database data inquiry request.
Optionally, after receiving the integration across database data inquiry request that user sends, method further include:
User query authority information is obtained according to user login information, to determine the corresponding inquiry qualifications of user；
Correlation inquiry information is parsed, determines the data table information in related each database, and be involved
Tables of data generate corresponding query statement, further comprise to obtain the query result of corresponding tables of data:
Correlation inquiry information is parsed, determines the data table information in related each database, and be involved
Tables of data generate it is corresponding comprising inquiry qualifications query statement, to obtain the query result of corresponding tables of data.
Optionally, correlation inquiry information is parsed, determines the data table information in related each database, and be
Related tables of data generates corresponding query statement, further comprises to obtain the query result of corresponding tables of data:
Correlation inquiry information is parsed, determines the data table information in related each database；Wherein, tables of data
Information includes table name, field and/or the querying condition that parsing obtains；
Corresponding query statement is generated according to data table information；
Query statement is executed, the query result of corresponding tables of data is obtained.
Optionally, after the query result for obtaining corresponding tables of data, method further include:
The field format for including in the query result for the tables of data that will acquire is converted to default specified format.
Optionally, the query result for being associated with each tables of data carries out secondary inquiry, obtains integration across database data inquiry request
Corresponding query result further comprises:
Correlation inquiry information is parsed, the incidence relation information of each tables of data of integration across database is obtained；
According to incidence relation information, the query statement for being associated with the query result of each tables of data is generated, to carry out secondary look into
It askes, obtains the corresponding query result of integration across database data inquiry request.
Optionally, after obtaining the corresponding query result of integration across database data inquiry request, method further include:
According to format as the result is shown specified in integration across database data inquiry request, to integration across database data inquiry request
Corresponding query result carries out format analysis processing；
The corresponding query result of integration across database data inquiry request after showing format analysis processing.
Optionally, method further include:
The corresponding query result of integration across database data inquiry request is saved into data file, for user's downloading；Its
In, the Table Header information of data file is arranged according to integration across database data inquiry request.
Optionally, method further include:
Data file is uploaded to cloud server.
Optionally, method further include:
When following abnormal conditions occur for monitoring, exception information is recorded in log, and carry out alarming processing；Wherein,
Abnormal conditions comprise determining that caused by correlation inquiry information it is abnormal, to correlation inquiry information parse it is generated it is abnormal,
It obtains abnormal, each tables of data of association query result caused by the query result of corresponding tables of data and carries out secondary inquiry institute
The exception of generation and/or query result is saved to abnormal caused by data file.
According to another aspect of the present invention, a kind of heterogeneous data source data acquisition facility is provided, is wrapped in heterogeneous data source
The database of multiple and different types is included, device includes:
Receiving module, the integration across database data inquiry request sent suitable for receiving user；
Relating module is suitable for determining the association of each tables of data of integration across database according to integration across database data inquiry request
Query information；
Enquiry module determines the tables of data letter in related each database suitable for parsing to correlation inquiry information
Breath, and corresponding query statement is generated for related tables of data, to obtain the query result of corresponding tables of data；According to association
Query information, the query result for being associated with each tables of data carry out secondary inquiry, it is corresponding to obtain integration across database data inquiry request
Query result.
Optionally, device further include:
Authority module is suitable for obtaining user query authority information according to user login information, be looked into determining that user is corresponding
Ask qualifications；
Enquiry module is further adapted for: being parsed to correlation inquiry information, is determined the number in related each database
The corresponding query statement comprising inquiry qualifications is generated according to table information, and for related tables of data, it is corresponding to obtain
The query result of tables of data.
Optionally, enquiry module is further adapted for:
Correlation inquiry information is parsed, determines the data table information in related each database；Wherein, tables of data
Information includes table name, field and/or the querying condition that parsing obtains；
Corresponding query statement is generated according to data table information；
Query statement is executed, the query result of corresponding tables of data is obtained.
Optionally, device further include:
Conversion module, the field format for including in the query result suitable for the tables of data that will acquire are converted to default specified lattice
Formula.
Optionally, enquiry module is further adapted for:
Correlation inquiry information is parsed, the incidence relation information of each tables of data of integration across database is obtained；
According to incidence relation information, the query statement for being associated with the query result of each tables of data is generated, to carry out secondary look into
It askes, obtains the corresponding query result of integration across database data inquiry request.
Optionally, device further include:
Display module, suitable for the format as the result is shown specified by integration across database data inquiry request, to across data
Data inquiry request corresponding query result in library carries out format analysis processing；Integration across database data inquiry request after showing format analysis processing
Corresponding query result.
Optionally, device further include:
Download module, suitable for saving the corresponding query result of integration across database data inquiry request into data file, with
It is downloaded for user；Wherein, the Table Header information of data file is arranged according to integration across database data inquiry request
Optionally, device further include:
Uploading module, suitable for data file is uploaded to cloud server.
Optionally, device further include:
Abnormal module is suitable for that exception information being recorded in log, and is accused when following abnormal conditions occur for monitoring
Alert processing；Wherein, abnormal conditions comprise determining that exception caused by correlation inquiry information, parse to correlation inquiry information
Abnormal, each tables of data of association query result caused by generated query result that is abnormal, obtaining corresponding tables of data
It carries out abnormal caused by secondary inquiry and/or saves query result to abnormal caused by data file.
According to another aspect of the invention, provide a kind of electronic equipment, comprising: processor, memory, communication interface and
Communication bus, processor, memory and communication interface complete mutual communication by communication bus；
Memory makes processor execute above-mentioned heterogeneous data source number for storing an at least executable instruction, executable instruction
According to the corresponding operation of acquisition methods.
In accordance with a further aspect of the present invention, a kind of computer storage medium is provided, at least one is stored in storage medium
Executable instruction, executable instruction make processor execute such as the corresponding operation of above-mentioned heterogeneous data source data capture method.
The heterogeneous data source data capture method and device provided according to the present invention receives the integration across database number that user sends
According to inquiry request；According to integration across database data inquiry request, the correlation inquiry information of each tables of data of integration across database is determined；It is right
Correlation inquiry information is parsed, and determines the data table information in related each database, and raw for related tables of data
At corresponding query statement, to obtain the query result of corresponding tables of data；According to correlation inquiry information, it is associated with each tables of data
Query result carry out secondary inquiry, obtain the corresponding query result of integration across database data inquiry request.According to the present invention it is possible to
User is helped to realize the data query of integration across database, so that user is not necessarily to pay close attention to the query language of disparate databases, data storage
Type not parity problem, obtains query result.
The above description is only an overview of the technical scheme of the present invention, in order to better understand the technical means of the present invention,
And it can be implemented in accordance with the contents of the specification, and in order to allow above and other objects of the present invention, feature and advantage can
It is clearer and more comprehensible, the followings are specific embodiments of the present invention.
Detailed description of the invention
By reading the following detailed description of the preferred embodiment, various other advantages and benefits are common for this field
Technical staff will become clear.The drawings are only for the purpose of illustrating a preferred embodiment, and is not considered as to the present invention
Limitation.And throughout the drawings, the same reference numbers will be used to refer to the same parts.In the accompanying drawings:
Fig. 1 shows the flow chart of heterogeneous data source data capture method according to an embodiment of the invention；
Fig. 2 shows the flow charts of heterogeneous data source data capture method in accordance with another embodiment of the present invention；
Fig. 3 shows the functional block diagram of heterogeneous data source data acquisition facility according to an embodiment of the invention；
Fig. 4 shows the structural schematic diagram of a kind of electronic equipment according to an embodiment of the invention.
Specific embodiment
Exemplary embodiments of the present disclosure are described in more detail below with reference to accompanying drawings.Although showing the disclosure in attached drawing
Exemplary embodiment, it being understood, however, that may be realized in various forms the disclosure without should be by embodiments set forth here
It is limited.On the contrary, these embodiments are provided to facilitate a more thoroughly understanding of the present invention, and can be by the scope of the present disclosure
It is fully disclosed to those skilled in the art.
Fig. 1 shows the flow chart of heterogeneous data source data capture method according to an embodiment of the invention.Such as Fig. 1 institute
Show, heterogeneous data source data capture method specifically comprises the following steps:
Step S101 receives the integration across database data inquiry request that user sends.
When user needs to carry out searching across data bases to the database of multiple and different types, i.e., heterogeneous data source is carried out
When inquiry, the integration across database data inquiry request that user sends is received.Wherein, inquiry request includes the inquiry item that user limits
Part and result information to be checked.
In the present embodiment in heterogeneous data source include multiple and different types database, as Explorer, Mysql, ODPS,
The different types of database such as HBase.Above data library type is for example, determining isomeric data with specific reference to performance
Database in source.There are query sentence of database format difference, data database storing type are different for distinct type data-base
The problems such as.The integration across database data inquiry request that user sends needs the database to multiple and different types to be associated inquiry.
The present embodiment makes the user do not need concern problem above, can directly obtain the query result of integration across database data inquiry request.
Step S102 determines the correlation inquiry of each tables of data of integration across database according to integration across database data inquiry request
Information.
Integration across database data inquiry request based on the received analyzes querying condition, result information to be checked, analysis
The tables of data of multiple databases, multiple databases involved by inquiry request.Such as basis in integration across database data inquiry request
XX querying condition will obtain XX result information to be checked.Database, data where field involved in analysis querying condition
Table, database, tables of data where field involved in result information to be checked.Meanwhile for querying condition and knot to be checked
Database involved in fruit information, tables of data, it is also necessary to be analyzed, obtain the incidence relation between tables of data.Specifically,
It needs to determine the tables of data of integration across database according to the mutual existing business association relationship of tables of data between multiple databases
Between inquiry incidence relation and single database in tables of data inquiry incidence relation.Between multiple databases present in tables of data
Business association relationship can be obtained by relation data between the table that pre-establishes, relation data and the practical execution phase of business between table
It closes.
In addition to database, tables of data involved in the querying condition and result information to be checked, also need to be related to it sometimes
His database, tables of data.Database, tables of data as involved in querying condition are A tables of data in AA database, result to be checked
Database, tables of data involved in information are B data table in BB database, between the two without direct incidence relation, are needed
B data table in A tables of data and BB database is connected in AA database by other intermediate databases, tables of data.Other numbers
It needs according to library, tables of data through each field in B data table in each field in A tables of data in analysis AA database, BB database, with
And tables of data between multiple databases existing business association relationship between each other, it both finds in other being respectively associated
Incidence relation between spatial database, tables of data and other databases, tables of data.Wherein database, tables of data may be for these
It is one or more.Integration across database data inquiry request is analyzed, obtains inquiry request and involved multiple data
Library, multiple databases tables of data.If without direct incidence relation between the tables of data of obtained multiple databases, it is also necessary to
According to the mutual existing business association relationship of tables of data between multiple databases, the tables of data of intermediate database is determined,
And field needed for the tables of data of intermediate database, querying condition etc..
By analyzing integration across database data inquiry request above, the correlation inquiry letter of each tables of data of integration across database is obtained
Breath.That is the correlation inquiry information of each tables of data of integration across database includes the tables of data, each of multiple databases, multiple databases
Inquiry incidence relation between the tables of data of field and integration across database that tables of data needs to inquire.
Step S103 parses correlation inquiry information, determines the data table information in related each database, and
Corresponding query statement is generated for related tables of data, to obtain the query result of corresponding tables of data.
Correlation inquiry information is parsed, the data table information in related each database is decoupled out.Wherein, data
Table information includes table name, field, the querying condition etc. that parsing obtains.The query statement of tables of data is generated according to data table information.
If including specific querying condition in data table information, when the field for needing to obtain, if the value of field a1 in A tables of data is 9, need
Field b1, c1 etc. are obtained, the corresponding query statement for generating A tables of data inquires field b1, c1 from A tables of data, inquires item
Part is A a1=9.If in data table information not including specific querying condition, it is whole that the query statement generated obtains tables of data
Information, query statement can be set as such as 1=1 without querying condition or by querying condition.In query statement inquire field according to
When data table information generation or not specified field, all fields of tables of data can be inquired.
It is that tables of data generates corresponding query statement according to data table information, executes and look into the affiliated database of the tables of data
Sentence is ask, the query result of corresponding tables of data is obtained.When generating query statement, need according to its affiliated type of database institute
The query statement rule used generates, and there is also differences for the tables of data of the disparate databases type query statement format to be generated.
After generating query statement according to type of database, query statement, the query result of the tables of data of acquisition are executed.The data that will acquire
The query result of table can first be stored in memory or specified address, facilitate subsequent use.For in related each database
Tables of data, generation query statement, the query result for obtaining each tables of data are saved.
Further, in the query result of the tables of data of acquisition each data type because type of database difference also can be different.?
After getting the query result of tables of data, for convenience of it is subsequent it is carried out using, it is also necessary to included to wherein each query result
Field format converted, be converted into default specified format.As converted in the following manner to it:
VARCHAR (" VARCHAR ", " VARCHAR ", String.class)；
LONGVARCHAR (" LONGVARCHAR ", " LONGVARCHAR ", String.class)；
CHAR (" CHAR ", " CHAR ", String.class)；
The above are for example, being the words such as CHAR, VARCHAR, LONGVARCHAR, TEXT by data type in conversion
The data of symbol form are converted to unified character string type, big byte data are converted to array type, such as by numerical value correlation
The data such as INT, INTEGER, TINYINT are converted to unified value type, by data such as bit data type BIT, Boolean types
Unified Boolean type is converted to, the data such as date data such as DATE, DATETIME, TIMESTAMP are converted into unified day
Phase type etc..
Step S104, according to correlation inquiry information, the query result for being associated with each tables of data carries out secondary inquiry, obtain across
The corresponding query result of database data inquiry request.
After the query result for obtaining each tables of data, it is also necessary to be associated with each tables of data according to correlation inquiry information.
Specifically, need to parse correlation inquiry information, obtain the incidence relation information of each tables of data of integration across database to get
To querying condition interconnected between each tables of data.According to incidence relation information, using the query result of each tables of data as
Sublist is inquired, the query statement for being associated with the query result of each tables of data is generated.Query statement is according to querying condition to each inquiry
Sublist is associated, and is executed to query statement, that is, carries out secondary inquiry, to obtain integration across database data inquiry request pair
The query result answered.
Further, after obtaining the corresponding query result of integration across database data inquiry request, according to user's transmission across number
According to library data inquiry request, it is also necessary to be shown query result, user is facilitated to check.Show when, can first according to across
Specified format as the result is shown, query result corresponding to integration across database data inquiry request in database data inquiry request
Carry out format analysis processing.It is 0.38 as data type is the data of floating point type in query result, specified format as the result is shown
When for percentage format, needs to be carried out format analysis processing, obtain 38%.Again by the integration across database data query after format analysis processing
It requests corresponding query result to show user, meets the needs of users.
The heterogeneous data source data capture method provided according to the present invention receives the integration across database data query that user sends
Request；According to integration across database data inquiry request, the correlation inquiry information of each tables of data of integration across database is determined；Association is looked into
It askes information to be parsed, determines the data table information in related each database, and generate and correspond to for related tables of data
Query statement, to obtain the query result of corresponding tables of data；According to correlation inquiry information, it is associated with the inquiry of each tables of data
As a result secondary inquiry is carried out, the corresponding query result of integration across database data inquiry request is obtained.According to the present invention it is possible to help to use
The data query of integration across database is realized at family, so that user is not necessarily to the query language for paying close attention to disparate databases, Stored Data Type not
Parity problem obtains query result.
Fig. 2 shows the flow charts of heterogeneous data source data capture method in accordance with another embodiment of the present invention.Such as Fig. 2
Shown, heterogeneous data source data capture method specifically comprises the following steps:
Step S201 receives the integration across database data inquiry request that user sends.
Step S202 determines the correlation inquiry of each tables of data of integration across database according to integration across database data inquiry request
Information.
Above step is with reference to the description of step S101-S102 in Fig. 1 embodiment, and details are not described herein.
Step S203 obtains user query authority information according to user login information, to determine the corresponding inquiry limit of user
Fixed condition.
When user needs integration across database to carry out data query, it is also necessary to consider the search access right problem of user.Inquiry power
The problems such as limit has ensured data safety, avoids user query to the data outside its search access right, data is caused to leak.User query
Authority information may insure that user only inquires the data in user query permission in inquiry.
The corresponding search access right of different user is different, according to user login information, such as user id, first determines looking into for user
Ask authority information.The corresponding inquiry qualifications of user can be determined according to the search access right information of user.Such as when user sends
Integration across database data inquiry request in when relating to user information inquiry, determine the corresponding inquiry qualifications of user only to look into
The user information of user id is ask, or only inquires the user information etc. of user id and user's id subordinate subscriber；When user sends
Integration across database data inquiry request in when relating to Transaction Information inquiry, determine the corresponding inquiry qualifications of user only to look into
The Transaction Information that the user id is created is ask, or only inquires the Transaction Information of the user id and the creation of user's id subordinate subscriber
Deng.The above are for example, the corresponding inquiry qualifications of specific search access right information are arranged according to performance.
Step S204 parses correlation inquiry information, determines the data table information in related each database, and
The corresponding query statement comprising inquiry qualifications is generated for related tables of data, to obtain the inquiry of corresponding tables of data
As a result.
Correlation inquiry information is parsed, after decoupling out the data table information in related each database, is generated each
The query statement of tables of data.
When user query qualifications are related to certain tables of data, when generating query statement, it is also necessary to limit the inquiry of user
Fixed condition is added in the query statement generated, and the search access right of user is exceeded to avoid the data of inquiry.Execute the inquiry generated
Sentence obtains the query result of tables of data corresponding with the search access right of user.
Step S205, according to correlation inquiry information, the query result for being associated with each tables of data carries out secondary inquiry, obtain across
The corresponding query result of database data inquiry request.
The description of the step step S104 of embodiment referring to Fig.1, details are not described herein.
Further, when generating query statement, the inquiry qualifications of user can also not included, be associated with each data
Include the inquiry qualifications of user when the query result of table carries out secondary inquiry, in the querying condition of secondary inquiry, ensures most
The query result obtained eventually meets the search access right of user.
Step S206 saves the corresponding query result of integration across database data inquiry request into data file, for
Family downloading.
The data that query result is included are written in data file and are saved, as being written in Excel file.It, can when write-in
To be write data into one by one according to query result.Existing written document technology can be used in specific written document, does not do expansion herein and says
It is bright.
It checks for the convenience of the user, the Table Header information in data file can be arranged according to integration across database data inquiry request.
Table Header information such as according to each field name of result to be checked in inquiry request as data file.
Further, query result being saved into data file can also need after showing query result further according to user
It asks and carries out generation data file, directly save to user and specify address；Data file first can also be uploaded to cloud server,
It is saved again to user when user needs to download and specifies address, when user being facilitated repeatedly to download, directly downloaded from cloud server,
Reduce the operation for the data write-in data file for being included by query result every time.
Further, in above-mentioned implementation procedure, the present embodiment further comprises abnormal monitoring processing.When monitoring generation is following different
When reason condition, exception information can be recorded in log, and carry out alarming processing.Abnormal conditions include as determined correlation inquiry
When information, when in the presence of correlation inquiry information can not be established it is generated it is abnormal, when being parsed to correlation inquiry information
Parsing etc. is abnormal, executes exception when obtaining the query result of corresponding tables of data, obtain it is abnormal etc. caused by it is abnormal, be associated with respectively
The query result of a tables of data generated when secondary inquiry it is various it is abnormal, produced when saving query result to data file
Raw write-in, preservation etc. are abnormal.When occurring above-mentioned abnormal, first the exception information of generation is recorded in journal file, and right
The carry out alarming processing answered.Alarming processing include such as mail alert, short message alarm, so as to timely and accurately to abnormal conditions into
Row processing.
The heterogeneous data source data capture method provided according to the present invention, inquiry when, according to the search access right of user into
Row inquiry, avoids leaking data.Meanwhile when obtaining query result, by the way that data file is written in query result, it can be convenient
The data of user's downloading heterogeneous data source.
Fig. 3 shows the functional block diagram of heterogeneous data source data acquisition facility according to an embodiment of the invention.Such as Fig. 3
Shown, heterogeneous data source data acquisition facility includes following module:
Receiving module 310 is suitable for: receiving the integration across database data inquiry request that user sends.
Relating module 320 is suitable for: according to integration across database data inquiry request, determining the pass of each tables of data of integration across database
Join query information.
Enquiry module 330 is suitable for: parsing to correlation inquiry information, determines the tables of data in related each database
Information, and corresponding query statement is generated for related tables of data, to obtain the query result of corresponding tables of data；According to pass
Join query information, the query result for being associated with each tables of data carries out secondary inquiry, and it is corresponding to obtain integration across database data inquiry request
Query result.
Optionally, device further include: authority module 340.
Authority module is suitable for: obtaining user query authority information according to user login information, is looked into determining that user is corresponding
Ask qualifications.
Enquiry module 330 is further adapted for: being parsed, is determined in related each database to correlation inquiry information
Data table information, and the corresponding query statement comprising inquiry qualifications is generated for related tables of data, to obtain correspondence
Tables of data query result.
Optionally, enquiry module 330 is further adapted for:
Correlation inquiry information is parsed, determines the data table information in related each database；Wherein, tables of data
Information includes table name, field and/or the querying condition that parsing obtains；Corresponding query statement is generated according to data table information；It holds
Row query statement obtains the query result of corresponding tables of data.
Optionally, device further include: conversion module 350.
Conversion module 350 is suitable for: the field format for including in the query result for the tables of data that will acquire is converted to default finger
Fix formula.
Optionally, enquiry module 330 is further adapted for: being parsed to correlation inquiry information, is obtained each of integration across database
The incidence relation information of a tables of data；According to incidence relation information, the inquiry language for being associated with the query result of each tables of data is generated
Sentence, to carry out secondary inquiry, obtains the corresponding query result of integration across database data inquiry request.
Optionally, device further include: display module 360.
Display module 360 is suitable for: according to format as the result is shown specified in integration across database data inquiry request, to across number
Format analysis processing is carried out according to the corresponding query result of library data inquiry request；Integration across database data query after showing format analysis processing is asked
Seek corresponding query result.
Optionally, device further include: download module 370.
Download module 370 is suitable for: the corresponding query result of integration across database data inquiry request saved into data file,
For user's downloading；Wherein, the Table Header information of data file is arranged according to integration across database data inquiry request
Optionally, device further include: uploading module 380.
Uploading module 380 is suitable for: data file is uploaded to cloud server.
Optionally, device further include: abnormal module 390.
Abnormal module 390 is suitable for: when following abnormal conditions occur for monitoring, exception information being recorded in log, is gone forward side by side
Row alarming processing；Wherein, abnormal conditions comprise determining that exception caused by correlation inquiry information, carry out to correlation inquiry information
Abnormal, each tables of data of association inquiry caused by query result that is abnormal caused by parsing, obtaining corresponding tables of data
As a result it carries out abnormal caused by secondary inquiry and/or saves query result to abnormal caused by data file.
Corresponding description in the description reference method embodiment of above each module, details are not described herein.
Present invention also provides a kind of nonvolatile computer storage media, the computer storage medium is stored at least
The heterogeneous data source data acquisition in above-mentioned any means embodiment can be performed in one executable instruction, the computer executable instructions
Method.
Fig. 4 shows the structural schematic diagram of a kind of electronic equipment according to an embodiment of the invention, and the present invention is specifically real
Example is applied not limit the specific implementation of electronic equipment.
As shown in figure 4, the electronic equipment may include: processor (processor) 402, communication interface
(Communications Interface) 404, memory (memory) 406 and communication bus 408.
Wherein:
Processor 402, communication interface 404 and memory 406 complete mutual communication by communication bus 408.
Communication interface 404, for being communicated with the network element of other equipment such as client or other servers etc..
Processor 402 can specifically execute above-mentioned heterogeneous data source data capture method and implement for executing program 410
Correlation step in example.
Specifically, program 410 may include program code, which includes computer operation instruction.
Processor 402 may be central processor CPU or specific integrated circuit ASIC (Application
Specific Integrated Circuit), or be arranged to implement the integrated electricity of one or more of the embodiment of the present invention
Road.The one or more processors that electronic equipment includes can be same type of processor, such as one or more CPU；It can also
To be different types of processor, such as one or more CPU and one or more ASIC.
Memory 406, for storing program 410.Memory 406 may include high speed RAM memory, it is also possible to further include
Nonvolatile memory (non-volatile memory), for example, at least a magnetic disk storage.
Program 410 specifically can be used for so that processor 402 executes the heterogeneous data source in above-mentioned any means embodiment
Data capture method.The specific implementation of each step may refer in above-mentioned heterogeneous data source data acquisition embodiment in program 410
Corresponding steps and unit in corresponding description, this will not be repeated here.It is apparent to those skilled in the art that being
Convenienct and succinct, the equipment of foregoing description and the specific work process of module of description, can be with reference in preceding method embodiment
Corresponding process description, details are not described herein.
Algorithm and display are not inherently related to any particular computer, virtual system, or other device provided herein.
Various general-purpose systems can also be used together with teachings based herein.As described above, it constructs required by this kind of system
Structure be obvious.In addition, the present invention is also not directed to any particular programming language.It should be understood that can use various
Programming language realizes summary of the invention described herein, and the description done above to language-specific is to disclose this hair
Bright preferred forms.
In the instructions provided here, numerous specific details are set forth.It is to be appreciated, however, that implementation of the invention
Example can be practiced without these specific details.In some instances, well known method, structure is not been shown in detail
And technology, so as not to obscure the understanding of this specification.
Similarly, it should be understood that in order to simplify the disclosure and help to understand one or more of the various inventive aspects,
Above in the description of exemplary embodiment of the present invention, each feature of the invention is grouped together into single implementation sometimes
In example, figure or descriptions thereof.However, the disclosed method should not be interpreted as reflecting the following intention: i.e. required to protect
Shield the present invention claims features more more than feature expressly recited in each claim.More precisely, such as right
As claim reflects, inventive aspect is all features less than single embodiment disclosed above.Therefore, it then follows tool
Thus claims of body embodiment are expressly incorporated in the specific embodiment, wherein each claim conduct itself
Separate embodiments of the invention.
Those skilled in the art will understand that can be carried out adaptively to the module in the equipment in embodiment
Change and they are arranged in one or more devices different from this embodiment.It can be the module or list in embodiment
Member or component are combined into a module or unit or component, and furthermore they can be divided into multiple submodule or subelement or
Sub-component.Other than such feature and/or at least some of process or unit exclude each other, it can use any
Combination is to all features disclosed in this specification (including adjoint claim, abstract and attached drawing) and so disclosed
All process or units of what method or apparatus are combined.Unless expressly stated otherwise, this specification is (including adjoint power
Benefit require, abstract and attached drawing) disclosed in each feature can carry out generation with an alternative feature that provides the same, equivalent, or similar purpose
It replaces.
In addition, it will be appreciated by those of skill in the art that although some embodiments described herein include other embodiments
In included certain features rather than other feature, but the combination of the feature of different embodiments mean it is of the invention
Within the scope of and form different embodiments.For example, in detail in the claims, embodiment claimed it is one of any
Can in any combination mode come using.
Various component embodiments of the invention can be implemented in hardware, or to run on one or more processors
Software module realize, or be implemented in a combination thereof.It will be understood by those of skill in the art that can be used in practice
Microprocessor or digital signal processor (DSP) realize heterogeneous data source data acquisition facility according to an embodiment of the present invention
In some or all components some or all functions.The present invention is also implemented as described herein for executing
Some or all device or device programs (for example, computer program and computer program product) of method.In this way
Realization program of the invention can store on a computer-readable medium, or can have the shape of one or more signal
Formula.Such signal can be downloaded from an internet website to obtain, and perhaps be provided on the carrier signal or with any other shape
Formula provides.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and ability
Field technique personnel can be designed alternative embodiment without departing from the scope of the appended claims.In the claims,
Any reference symbol between parentheses should not be configured to limitations on claims.Word "comprising" does not exclude the presence of not
Element or step listed in the claims.Word "a" or "an" located in front of the element does not exclude the presence of multiple such
Element.The present invention can be by means of including the hardware of several different elements and being come by means of properly programmed computer real
It is existing.In the unit claims listing several devices, several in these devices can be through the same hardware branch
To embody.The use of word first, second, and third does not indicate any sequence.These words can be explained and be run after fame
Claim.
Claims (10)
1. a kind of heterogeneous data source data capture method, wherein include the data of multiple and different types in the heterogeneous data source
Library, method include:
Receive the integration across database data inquiry request that user sends；
According to the integration across database data inquiry request, the correlation inquiry information of each tables of data of integration across database is determined；
The correlation inquiry information is parsed, determines the data table information in related each database, and be involved
Tables of data generate corresponding query statement, to obtain the query result of corresponding tables of data；According to the correlation inquiry information,
The query result for being associated with each tables of data carries out secondary inquiry, obtains the corresponding inquiry knot of the integration across database data inquiry request
Fruit.
2. according to the method described in claim 1, wherein, the integration across database data inquiry request that the reception user sends it
Afterwards, the method also includes:
User query authority information is obtained according to user login information, to determine the corresponding inquiry qualifications of user；
It is described that the correlation inquiry information is parsed, determine the data table information in related each database, and be institute
The tables of data being related to generates corresponding query statement, further comprises to obtain the query result of corresponding tables of data:
The correlation inquiry information is parsed, determines the data table information in related each database, and be involved
Tables of data generate it is corresponding comprising it is described inquiry qualifications query statement, to obtain the inquiry knot of corresponding tables of data
Fruit.
3. described to be parsed to the correlation inquiry information according to the method described in claim 1, wherein, determine involved by
Each database in data table information, and corresponding query statement is generated for related tables of data, to obtain corresponding number
Further comprise according to the query result of table:
The correlation inquiry information is parsed, determines the data table information in related each database；Wherein, the number
It include table name, field and/or the querying condition that parsing obtains according to table information；
Corresponding query statement is generated according to the data table information；
The query statement is executed, the query result of corresponding tables of data is obtained.
4. method according to any one of claim 1-3, wherein in the query result for obtaining corresponding tables of data
Later, the method also includes:
The field format for including in the query result for the tables of data that will acquire is converted to default specified format.
5. it is described according to the correlation inquiry information according to the method described in claim 1, wherein, it is associated with each tables of data
Query result carries out secondary inquiry, and obtaining the corresponding query result of the integration across database data inquiry request further comprises:
The correlation inquiry information is parsed, the incidence relation information of each tables of data of integration across database is obtained；
According to the incidence relation information, the query statement for being associated with the query result of each tables of data is generated, to carry out secondary look into
It askes, obtains the corresponding query result of the integration across database data inquiry request.
6. method according to any one of claims 1-5, wherein obtain the integration across database data query described and ask
After seeking corresponding query result, the method also includes:
According to format as the result is shown specified in the integration across database data inquiry request, to the integration across database data query
Corresponding query result is requested to carry out format analysis processing；
The corresponding query result of the integration across database data inquiry request after showing format analysis processing.
7. method according to claim 1 to 6, wherein the method also includes:
The corresponding query result of the integration across database data inquiry request is saved into data file, for user's downloading；Its
In, the Table Header information of the data file is arranged according to the integration across database data inquiry request.
8. a kind of heterogeneous data source data acquisition facility, wherein include the data of multiple and different types in the heterogeneous data source
Library, device include:
Receiving module, the integration across database data inquiry request sent suitable for receiving user；
Relating module is suitable for determining the association of each tables of data of integration across database according to the integration across database data inquiry request
Query information；
Enquiry module determines the tables of data letter in related each database suitable for parsing to the correlation inquiry information
Breath, and corresponding query statement is generated for related tables of data, to obtain the query result of corresponding tables of data；According to described
Correlation inquiry information, the query result for being associated with each tables of data carry out secondary inquiry, obtain the integration across database data query and ask
Seek corresponding query result.
9. a kind of electronic equipment, comprising: processor, memory, communication interface and communication bus, the processor, the storage
Device and the communication interface complete mutual communication by the communication bus；
The memory executes the processor as right is wanted for storing an at least executable instruction, the executable instruction
Ask the corresponding operation of heterogeneous data source data capture method described in any one of 1-7.
10. a kind of computer storage medium, an at least executable instruction, the executable instruction are stored in the storage medium
Processor is set to execute such as the corresponding operation of heterogeneous data source data capture method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910465165.9A CN110162544B (en) | 2019-05-30 | 2019-05-30 | Heterogeneous data source data acquisition method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910465165.9A CN110162544B (en) | 2019-05-30 | 2019-05-30 | Heterogeneous data source data acquisition method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110162544A true CN110162544A (en) | 2019-08-23 |
| CN110162544B CN110162544B (en) | 2022-05-27 |
Family
ID=67630634
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910465165.9A Active CN110162544B (en) | 2019-05-30 | 2019-05-30 | Heterogeneous data source data acquisition method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110162544B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110955662A (en) * | 2019-11-29 | 2020-04-03 | 车智互联（北京）科技有限公司 | Method, computing device and storage medium for maintaining data table association relation |
| CN111104426A (en) * | 2019-11-22 | 2020-05-05 | 深圳智链物联科技有限公司 | Data query method and system |
| CN111177213A (en) * | 2019-12-16 | 2020-05-19 | 北京淇瑀信息科技有限公司 | Privacy cluster self-service query platform and method and electronic equipment |
| CN111259038A (en) * | 2020-01-16 | 2020-06-09 | 北京思特奇信息技术股份有限公司 | Database query and data export method, system, medium and equipment |
| CN111259036A (en) * | 2020-01-10 | 2020-06-09 | 苏州达家迎信息技术有限公司 | Cross-library and cross-table query method, device, server and storage medium |
| CN111581231A (en) * | 2020-04-20 | 2020-08-25 | 北京明略软件***有限公司 | Query method and device based on heterogeneous database |
| WO2021129498A1 (en) * | 2019-12-24 | 2021-07-01 | 阿里巴巴集团控股有限公司 | Data processing method and apparatus for distributed query system |
| CN113946594A (en) * | 2021-12-22 | 2022-01-18 | 昆仑智汇数据科技(北京)有限公司 | Integrated query method, device and equipment for industrial multi-source heterogeneous data |
| CN114490241A (en) * | 2021-12-25 | 2022-05-13 | 苏州浪潮智能科技有限公司 | Chip monitoring method, system, storage medium and equipment |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120166424A1 (en) * | 2010-10-26 | 2012-06-28 | ParElastic Corporation | Apparatus for Elastic Database Processing with Heterogeneous Data |
| CN104239320A (en) * | 2013-06-14 | 2014-12-24 | 深圳中兴网信科技有限公司 | Data merging method and system |
| CN104573115A (en) * | 2015-02-04 | 2015-04-29 | 新余兴邦信息产业有限公司 | Method and system for achieving integration interface supporting operation of multi-type databases |
| CN105224613A (en) * | 2015-09-17 | 2016-01-06 | 西安未来国际信息股份有限公司 | Based on integrating heterogeneous data source system and the integration method thereof of the federal technology of data |
| US20160098485A1 (en) * | 2014-10-05 | 2016-04-07 | Splunk Inc. | Field Value Search Drill Down |
| US20160306852A1 (en) * | 2015-03-11 | 2016-10-20 | International Business Machines Corporation | Answering natural language table queries through semantic table representation |
| CN106372177A (en) * | 2016-08-30 | 2017-02-01 | 东华大学 | Query expansion method supporting correlated query and fuzzy grouping of mixed data type |
| CN107066499A (en) * | 2016-12-30 | 2017-08-18 | 江苏瑞中数据股份有限公司 | The data query method of multi-source data management and visualization system is stored towards isomery |
| CN107491510A (en) * | 2017-08-03 | 2017-12-19 | 国网江苏省电力公司信息通信分公司 | One kind mixing heterogeneous data source unified query system and distributed enquiring method |
| CN108509637A (en) * | 2018-04-10 | 2018-09-07 | 口碑(上海)信息技术有限公司 | Tables of data relation query method and device |
| CN108804460A (en) * | 2017-05-03 | 2018-11-13 | 北京润乾信息***技术有限公司 | A kind of query language based on SQL |
-
2019
- 2019-05-30 CN CN201910465165.9A patent/CN110162544B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120166424A1 (en) * | 2010-10-26 | 2012-06-28 | ParElastic Corporation | Apparatus for Elastic Database Processing with Heterogeneous Data |
| CN104239320A (en) * | 2013-06-14 | 2014-12-24 | 深圳中兴网信科技有限公司 | Data merging method and system |
| US20160098485A1 (en) * | 2014-10-05 | 2016-04-07 | Splunk Inc. | Field Value Search Drill Down |
| CN104573115A (en) * | 2015-02-04 | 2015-04-29 | 新余兴邦信息产业有限公司 | Method and system for achieving integration interface supporting operation of multi-type databases |
| US20160306852A1 (en) * | 2015-03-11 | 2016-10-20 | International Business Machines Corporation | Answering natural language table queries through semantic table representation |
| CN105224613A (en) * | 2015-09-17 | 2016-01-06 | 西安未来国际信息股份有限公司 | Based on integrating heterogeneous data source system and the integration method thereof of the federal technology of data |
| CN106372177A (en) * | 2016-08-30 | 2017-02-01 | 东华大学 | Query expansion method supporting correlated query and fuzzy grouping of mixed data type |
| CN107066499A (en) * | 2016-12-30 | 2017-08-18 | 江苏瑞中数据股份有限公司 | The data query method of multi-source data management and visualization system is stored towards isomery |
| CN108804460A (en) * | 2017-05-03 | 2018-11-13 | 北京润乾信息***技术有限公司 | A kind of query language based on SQL |
| CN107491510A (en) * | 2017-08-03 | 2017-12-19 | 国网江苏省电力公司信息通信分公司 | One kind mixing heterogeneous data source unified query system and distributed enquiring method |
| CN108509637A (en) * | 2018-04-10 | 2018-09-07 | 口碑(上海)信息技术有限公司 | Tables of data relation query method and device |
Non-Patent Citations (3)
| Title |
|---|
| 杨寒光 等: ""分布式异构数据源标准化查询设计与实现"", 《土木建筑工程信息技术》 * |
| 杨寒光 等: ""分布式异构数据源标准化查询设计与实现"", 《土木建筑工程信息技术》, vol. 5, no. 4, 31 August 2013 (2013-08-31), pages 61 - 63 * |
| 郭超: ""基于模式映射的异构数据源集成查询关键技术研究"", 《科技信息》, 5 April 2012 (2012-04-05), pages 234 * |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111104426A (en) * | 2019-11-22 | 2020-05-05 | 深圳智链物联科技有限公司 | Data query method and system |
| CN111104426B8 (en) * | 2019-11-22 | 2024-04-23 | 北京傲速科技有限公司 | Data query method and system |
| CN111104426B (en) * | 2019-11-22 | 2024-04-05 | 深圳智链物联科技有限公司 | Data query method and system |
| CN110955662A (en) * | 2019-11-29 | 2020-04-03 | 车智互联（北京）科技有限公司 | Method, computing device and storage medium for maintaining data table association relation |
| CN111177213A (en) * | 2019-12-16 | 2020-05-19 | 北京淇瑀信息科技有限公司 | Privacy cluster self-service query platform and method and electronic equipment |
| CN111177213B (en) * | 2019-12-16 | 2024-04-19 | 北京淇瑀信息科技有限公司 | Privacy cluster self-service query platform, method and electronic equipment |
| WO2021129498A1 (en) * | 2019-12-24 | 2021-07-01 | 阿里巴巴集团控股有限公司 | Data processing method and apparatus for distributed query system |
| CN111259036B (en) * | 2020-01-10 | 2022-10-11 | 苏州达家迎信息技术有限公司 | Cross-library and cross-table query method, device, server and storage medium |
| CN111259036A (en) * | 2020-01-10 | 2020-06-09 | 苏州达家迎信息技术有限公司 | Cross-library and cross-table query method, device, server and storage medium |
| CN111259038A (en) * | 2020-01-16 | 2020-06-09 | 北京思特奇信息技术股份有限公司 | Database query and data export method, system, medium and equipment |
| CN111581231A (en) * | 2020-04-20 | 2020-08-25 | 北京明略软件***有限公司 | Query method and device based on heterogeneous database |
| CN113946594B (en) * | 2021-12-22 | 2022-07-12 | 昆仑智汇数据科技(北京)有限公司 | Integrated query method, device and equipment for industrial multi-source heterogeneous data |
| CN113946594A (en) * | 2021-12-22 | 2022-01-18 | 昆仑智汇数据科技(北京)有限公司 | Integrated query method, device and equipment for industrial multi-source heterogeneous data |
| CN114490241B (en) * | 2021-12-25 | 2023-09-15 | 苏州浪潮智能科技有限公司 | Chip monitoring method, system, storage medium and device |
| CN114490241A (en) * | 2021-12-25 | 2022-05-13 | 苏州浪潮智能科技有限公司 | Chip monitoring method, system, storage medium and equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110162544B (en) | 2022-05-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110162544A (en) | Heterogeneous data source data capture method and device | |
| CN106778288B (en) | A kind of method and system of data desensitization | |
| US8584112B2 (en) | Open application lifecycle management framework | |
| CN110032575A (en) | Data query method, apparatus, equipment and storage medium | |
| CN107257973B (en) | Querying data sources on a network | |
| CN111639078A (en) | Data query method and device, electronic equipment and readable storage medium | |
| US20100100872A1 (en) | Methods and systems for implementing a test automation framework for testing software applications on unix/linux based machines | |
| CN111309760A (en) | Data retrieval method, system, device and storage medium | |
| US9391825B1 (en) | System and method for tracking service results | |
| CN110688544A (en) | Method, device and storage medium for querying database | |
| CN112860730A (en) | SQL statement processing method and device, electronic equipment and readable storage medium | |
| US9262474B2 (en) | Dynamic domain query and query translation | |
| CN113760948A (en) | Data query method and device | |
| CN109992500A (en) | Interface test method, device and the computer readable storage medium of application | |
| US11704327B2 (en) | Querying distributed databases | |
| US11513876B2 (en) | Resolving data location for queries in a multi-system instance landscape | |
| US11693859B2 (en) | Systems and methods for data retrieval from a database indexed by an external search engine | |
| CN110674177B (en) | Data query method and device, electronic equipment and storage medium | |
| CN113760961A (en) | Data query method and device | |
| US20220334817A1 (en) | Cross platform configuration domain translation | |
| CN110472127A (en) | A kind of data query method and system | |
| US10628416B2 (en) | Enhanced database query processing | |
| CN114341833B (en) | Reducing TEMP size on base table | |
| CN114020789A (en) | Bill data query method, system, device and storage medium | |
| CN108629003B (en) | Content loading method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |