CN107818127A - A kind of querying method and system for multi-source data - Google Patents
A kind of querying method and system for multi-source data Download PDFInfo
- Publication number
- CN107818127A CN107818127A CN201710808772.1A CN201710808772A CN107818127A CN 107818127 A CN107818127 A CN 107818127A CN 201710808772 A CN201710808772 A CN 201710808772A CN 107818127 A CN107818127 A CN 107818127A
- Authority
- CN
- China
- Prior art keywords
- data
- source
- database
- request
- access request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 32
- AFYCEAFSNDLKSX-UHFFFAOYSA-N coumarin 460 Chemical compound CC1=CC(=O)OC2=CC(N(CC)CC)=CC=C21 AFYCEAFSNDLKSX-UHFFFAOYSA-N 0.000 claims abstract description 37
- 238000012546 transfer Methods 0.000 claims abstract description 16
- 238000007726 management method Methods 0.000 claims description 51
- 238000012545 processing Methods 0.000 claims description 12
- 230000004044 response Effects 0.000 claims description 6
- 238000012550 audit Methods 0.000 claims description 5
- 239000000284 extract Substances 0.000 claims description 5
- 230000004913 activation Effects 0.000 claims description 4
- 238000012423 maintenance Methods 0.000 claims description 4
- 230000009897 systematic effect Effects 0.000 claims description 4
- 238000004458 analytical method Methods 0.000 claims description 3
- 238000013480 data collection Methods 0.000 claims description 3
- 238000011156 evaluation Methods 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims 1
- 238000013523 data management Methods 0.000 abstract description 5
- 230000010354 integration Effects 0.000 abstract description 4
- 230000006870 function Effects 0.000 description 7
- 230000008569 process Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 238000000605 extraction Methods 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 2
- 238000013475 authorization Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 206010016256 fatigue Diseases 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006116 polymerization reaction Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/256—Integrating or interfacing systems involving database management systems in federated or virtual databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2452—Query translation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/52—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems during program execution, e.g. stack integrity ; Preventing unwanted data erasure; Buffer overflow
- G06F21/54—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems during program execution, e.g. stack integrity ; Preventing unwanted data erasure; Buffer overflow by adding security routines or objects to programs
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computer Security & Cryptography (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- Fuzzy Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a kind of querying method and system for multi-source data, belong to database field, data access request including receiving service application, is forwarded to MDAC, MDAC is judged the Transfer Parameters in access request by access request；When it is inter-library request of data that judged result, which shows data access, data access request is forwarded to distributed data query engine；Multi-data source corresponding with data access request is connected to, Query Result corresponding with data access request is obtained at multi-data source, Query Result is forwarded to MDAC so that data forwarding component sends Query Result to service application.By establishing a database in logic, isolate between service application and database and be directly connected to, so as to improve the security of data access；Abstract exploitation service is provided, simplifies the exploitation of data management and application；The correlation inquiry between physically inter-library table is supported, realizes the multi-source data inquiry of integration across database.
Description
Technical field
The invention belongs to database field, more particularly to a kind of querying method and system for multi-source data.
Background technology
With cloud computing, the continuous development of big data technology, the information-based water as the power industry of national pillar industry
Flat also to improve constantly, each business core system and data central product have tired out the data of magnanimity, therefore how effectively to inquire about simultaneously
Application data turns into an important topic of electric power trade informationization research.
Current data query is based primarily upon SQL (Structured Query Language, SQL) and entered
OK, but the query process can only often be directed to one or single datum storehouse and carry out, for broad categories, large number of inter-library
Data query, it can not often accomplish accurately to inquire about, so as to reduce the validity of data query.
The content of the invention
In order to solve shortcoming and defect present in prior art, the invention provides establish logical data base so as to looking into
Ask request and carry out classification processing, the final querying method and system for multi-source data for improving data query validity.
In order to reach above-mentioned technical purpose, the invention provides a kind of querying method for multi-source data, the inquiry
Method is used to provide data access response between service application layer and database, the querying method, including：
The data access request from service application is received, access request is forwarded to MDAC, data access
Transfer Parameters are judged by the Transfer Parameters in component extraction access request；
When it is inter-library request of data that judged result, which shows data access, MDAC is made to turn data access request
It is sent to distributed data query engine；
Make distributed data query engine be connected to multi-data source corresponding with data access request, obtained at multi-data source
Query Result corresponding with data access request is taken, Query Result is forwarded to MDAC so that data forwarding component
Query Result is sent to service application.
Optionally, the form of expression of the data access request is SQL.
Optionally, the MDAC is offer standard java Java DataBase Connections, docked with service application
Interface.
Optionally, the MDAC, is used for：
Create connection and send SQL request；
Wherein, connection, including correct configuration JDBC parameters are created, JDBC parameters are transmitted to connection management module, connecting tube
Reason template provides parameter to state server according to JDBC versions and verified, obtains metadata information；
SQL request, including whether the SQL connections for judging to generate are to penetrate connection, if connection is penetrated, then based on given
Physical database create or select on demand connection.
On the other hand, the embodiment of the present application additionally provides a kind of inquiry system for multi-source data, the querying method
For providing data access response between service application layer and database, it is characterised in that the inquiry system, including：
Service access application programming interfaces, for receiving the data access request from service application, access request is turned
MDAC is sent to, MDAC extracts the Transfer Parameters in access request, Transfer Parameters are judged；
JDBC HUB engine container modules, for when it is inter-library request of data that judged result, which shows data access, making number
Data access request is forwarded to distributed data query engine according to component is accessed；
Application database drives layer, corresponding with data access request more for making distributed data query engine be connected to
Data source, Query Result corresponding with data access request is obtained at multi-data source；
Service access application programming interfaces, it is additionally operable to Query Result being forwarded to MDAC so that data forwarding
Component sends Query Result to service application.
Optionally, the service access application programming interfaces, are specifically used for：
JDBC agreement access interfaces are provided, capture the command request that application database driving is sent, to operation system and
The transparent unaware of Service Database, support multiple database hybrid protocol；
The parameter transmitted by SQL connections judges need which kind of types of database accessed, and is parsed, analyzed and processed.
Optionally, the JDBC HUB engines container module, including：
SQL parsings, data route, driven management, data source control, performance evaluation, operation audit, data parsing and place
Reason, SQL performance data collections, rights management and storehouse table management；
Wherein, rights management includes carrying out storehouse, the other rights management of table level and control, and fine-grained authority relies on bottom number
Controlled according to source；
Driven management includes the various activation bits of user's registration, supports the management of multi version driving.
Instance management includes the various systematic parameters for being supplied to platform operation maintenance personnel configuration management JDBC HUB examples.
Optionally, the SQL parsings, including：
The request analysis of application is distributed to specific data source for SQL and prepared to relevant informations such as storehouse, tables.Finally please
Ask and the adapter by relevant database is connected to each relevant database.
Optionally, the data route, including：
Data are route：According to the SQL results parsed and the information of storehouse table name word space management, it is determined which data source used
Adapter and final data source, and establish connection and request is forwarded, for the raising of performance, support connection pool etc. and excellent
Change technology, while be the access point of various processing plug-in units, such as performance collection module.
Optionally, the MDAC, is used for：
Create connection and send SQL request；
Wherein, connection, including correct configuration JDBC parameters are created, JDBC parameters are transmitted to connection management module, connecting tube
Reason template provides parameter to state server according to JDBC versions and verified, obtains metadata information；
SQL request, including whether the SQL connections for judging to generate are to penetrate connection, if connection is penetrated, then based on given
Physical database create or select on demand connection.
The beneficial effect that technical scheme provided by the invention is brought is：
By establishing a database in logic, isolate between service application and database and be directly connected to, so as to improve
The security of data access；Abstract exploitation service is provided, simplifies the exploitation of data management and application；Support physically inter-library
Correlation inquiry between table, realize the multi-source data inquiry of integration across database.
Brief description of the drawings
In order to illustrate more clearly of technical scheme, the required accompanying drawing used in being described below to embodiment
It is briefly described, it should be apparent that, drawings in the following description are only some embodiments of the present invention, general for this area
For logical technical staff, on the premise of not paying creative work, other accompanying drawings can also be obtained according to these accompanying drawings.
Fig. 1 is a kind of schematic flow sheet of querying method for multi-source data provided by the invention；
Fig. 2 is a kind of schematic flow sheet of querying method for multi-source data provided by the invention；
Fig. 3 is a kind of schematic flow sheet of querying method for multi-source data provided by the invention.
Embodiment
To make the structure of the present invention and advantage clearer, the structure of the present invention is made further below in conjunction with accompanying drawing
Description.
Embodiment one
The invention provides a kind of querying method for multi-source data, the querying method be used for service application layer with
Data access response is provided between database, as shown in figure 1, the querying method, including：
11st, the data access request from service application is received, access request is forwarded to MDAC, data are visited
The Transfer Parameters in component extraction access request are asked, Transfer Parameters are judged；
12nd, when it is inter-library request of data that judged result, which shows data access, make MDAC please by data access
Ask and be forwarded to distributed data query engine；
13rd, distributed data query engine is made to be connected to multi-data source corresponding with data access request, from multi-data source
Place obtains Query Result corresponding with data access request, Query Result is forwarded into MDAC so that data forwarding
Component sends Query Result to service application.
In force, multi-source data querying method provided in an embodiment of the present invention, committed step are to establish a logic
On database, isolate between service application and database and be directly connected to, so as to improve the security of data access；There is provided abstract
Exploitation service, simplify the exploitation of data management and application；The correlation inquiry between physically inter-library table is supported, is realized across number
Inquired about according to the multi-source data in storehouse.
In specific implementation, the querying method builds multi-source using multi-source data Connection inquiring engine as technical support
Data query function.The requirement source of multi-source data inquiry is in multiple tables of multiple Database Systems, due to different types of data
The database semantic and query grammar in storehouse are had any different to be connect, it is necessary to build the programming of data, services access application in query facility
Mouthful, by SQL (Structured Query Language, SQL) parse, data route, data parsing and
The functions such as processing realize that by front end data inquiry operation be the database semantic of target database and inquiry language by logical transition
Method, and be forwarded to corresponding target data storehouse in the form of request and perform data query and return to result.
In addition, querying method proposed by the present invention, can also be real by using the visual development interface of What You See Is What You Get
Existing storehouse table polymerization, field conversion, i.e., it need to screen and carry out dynamic construction and go out Dan Ku, inter-library data application, its compatible industry is a variety of
Data source (such as main flow relational database, distributed data base), data mining person without focused data ETL can low threshold enter
Row data application and development.(Extract-Transform-Load, for describing data from source terminal by extracting
(extract) (transform), the process of loading (load) to destination, are changed.)
Realize that logic is：
Application system java databases connection (Java DataBase Connectivity, JDBC) communication packet is captured,
The data of different business systems operation are analyzed,
Asked by certain rule and adaptation is forwarded to each specific business datum source, because all business datums please
Ask and all carried out by JDBC HUB, the data application such as follow-up data duplication, operation monitoring, encryption just has unified extension
Platform.
Upper using, by JDBC Hub, application program can use known JDBC interfaces establish data storage and calculate
Service and the connections of Java data-applications, and the table, view and storing process that bottom data source can be extracted be used for retrieving and
Update the data.
Wherein, above-mentioned JDBC HUB (Java DataBase Connectivity HUB, Java database connection forwardings
Device) it is intended that various JDBC data sources provide unified outbound data service, make lower data storage computing resource it is unified,
Conducted interviews under controlled mode.So that it is directly connected to be decoupled using between the specific data source of bottom.
Optionally, the form of expression of the data access request is SQL.
In force, SQL (Structured Query Language) abbreviation SQL, it is a kind of special
The programming language of purpose, it is a kind of data base querying and programming language, for accessing data and inquiry, renewal and management
Relational database system；It is also the extension name of database script file simultaneously.
Optionally, the MDAC is offer standard java Java DataBase Connections, docked with service application
Interface.
The MDAC, is used for：
Create connection and send SQL request；
Wherein, connection, including correct configuration JDBC parameters are created, JDBC parameters are transmitted to connection management module, connecting tube
Reason template provides parameter to state server according to JDBC versions and verified, obtains metadata information；
SQL request, including whether the SQL connections for judging to generate are to penetrate connection, if connection is penetrated, then based on given
Physical database create or select on demand connection.
In force, MDAC is as the unique interface docked with service application layer, for for java databases
The driving of offer standard is provided.The component is embedded in the inside of the specific module of service application, realizes the initialization of application authorization process,
The Notification of Changes for obtaining the metadata information needed for application authorization and being sent according to metadata management carries out change operation in real time.
The invention provides a kind of querying method for multi-source data, including receive the data access from service application
Request, access request is forwarded to MDAC, MDAC extracts the Transfer Parameters in access request, to transmitting
Parameter is judged；When it is inter-library request of data that judged result, which shows data access, MDAC is made by data access
Request is forwarded to distributed data query engine；Distributed data query engine is made to be connected to corresponding with data access request more
Data source, Query Result corresponding with data access request is obtained at multi-data source, Query Result is forwarded to data access
Component so that data forwarding component sends Query Result to service application.By establishing a database in logic, isolation
It is directly connected between service application and database, so as to improve the security of data access；Abstract exploitation service is provided, simplified
Data management and the exploitation of application；The correlation inquiry between physically inter-library table is supported, realizes the multi-source data of integration across database
Inquiry.
Embodiment two
The embodiment of the present application additionally provides a kind of inquiry system 2 for multi-source data, and the querying method is used in industry
Data access response is provided between application layer of being engaged in and database, as shown in Fig. 2 the inquiry system 2, including：
Service access application programming interfaces 21, for receiving the data access request from service application, by access request
MDAC is forwarded to, MDAC extracts the Transfer Parameters in access request, Transfer Parameters are judged；
JDBC HUB engine containers module 22, for when it is inter-library request of data that judged result, which shows data access, making
Data access request is forwarded to distributed data query engine by MDAC；
Application database drives layer 23, corresponding with data access request for making distributed data query engine be connected to
Multi-data source, Query Result corresponding with data access request is obtained at multi-data source；
Service access application programming interfaces 21, it is additionally operable to Query Result being forwarded to MDAC so that data turn
Hair component sends Query Result to service application.
In force, the suggestion of this inquiry system utilizes the open storage computing architecture in big data platform assembly, to determine
Adopted JDBC HUB engines container template, JDBC HUB examples are then built and manage to carry out function realization.
Realize that logic is：
Application system JDBC communication packets are captured, analyze the data of different business systems operation,
Asked by certain rule and adaptation is forwarded to each specific business datum source, because all business datums please
Ask and all carried out by JDBC HUB, the data application such as follow-up data duplication, operation monitoring, encryption just has unified extension
Platform.
Upper using, by JDBC Hub, application program can use known JDBC interfaces establish data storage and calculate
Service and the connections of Java data-applications, and the table, view and storing process that bottom data source can be extracted be used for retrieving and
Update the data.
Optionally, the service access application programming interfaces, are specifically used for：
JDBC agreement access interfaces are provided, capture the command request that application database driving is sent, to operation system and
The transparent unaware of Service Database, support multiple database hybrid protocol；
The parameter transmitted by SQL connections judges need which kind of types of database accessed, and is parsed, analyzed and processed.
In force, service access application programming interfaces are service access API：It is main that JDBC agreement access interfaces are provided,
It can capture the command request that application end database-driven (Oracle MySQL PostgreSQL clients) is sent, to industry
Business system and the transparent unaware of Service Database, support multiple database hybrid protocol.Its parameter transmitted by SQL connections judges
Need which kind of types of database accessed, then parsed, analyzed and processed.
Optionally, the JDBC HUB engines container module, including：
SQL parsings, data route, driven management, data source control, performance evaluation, operation audit, data parsing and place
Reason, SQL performance data collections, rights management and storehouse table management；
Wherein, rights management includes carrying out storehouse, the other rights management of table level and control, and fine-grained authority relies on bottom number
Controlled according to source；
Driven management includes the various activation bits of user's registration, supports the management of multi version driving.
Instance management includes the various systematic parameters for being supplied to platform operation maintenance personnel configuration management JDBC HUB examples.
In force, as shown in figure 3, JDBC HUB engine containers module includes numerous function sub-modules, including
SQL is parsed：The request analysis of application is distributed to specific data source for SQL and does standard to relevant informations such as storehouse, tables
It is standby.Finally ask the adapter by relevant database being connected to each relevant database.
Data are route：According to the SQL results parsed and the information of storehouse table name word space management, it is determined which data source used
Adapter and final data source, and establish connection and request is forwarded, for the raising of performance, support connection pool etc. and excellent
Change technology, while be the access point of various processing plug-in units, such as performance collection module.
Data parse and processing：Extra processing is carried out to the request data and returning result for being sent to data source adapter
Basic framework, it may be inserted into various processing modules and carry out two-way extra processing, such as carry out the support of client definition function, property
The access, the filtering of security module etc. of energy module.
Storehouse table management：There is provided unified storehouse table name word space management (mapping in actual storehouse, table to platform interior storehouse table,
Global naming space) and table define mapping management, and provide mapping relations of the storehouse to actual physics node, database
The information such as type, existing state, principal and subordinate's mark.When the request that no storehouse table defines accesses, it is directly connected to using direct-connected mode
To the data source of rear end.
Data source control：The various bottom data sources of user's registration, it is responsible for final and bottom layer driving interaction.The pond of connection
And fault-tolerant also it is supported in module.Using can be with predefined fault-tolerant processing mechanism, when bottom connection error occurs for request
When, fault-tolerant processing is carried out, the other data sources of bottom is changed, carries out retrying operation.
Rights management：Storehouse, the other rights management of table level and control are carried out, fine-grained authority relies on bottom data source to control
System.
Driven management：The various activation bits of user's registration, support the management of multi version driving.
Instance management：It is supplied to the various systematic parameters of platform operation maintenance personnel configuration management JDBC HUB examples.
Deployment management：The management of the deployment template of the related each first floor system of this module is provided, facilitates the dynamic of Service Instance
State extends and upgrading, this function need the deployment template platform outside cooperation to be supported.
Performance monitoring：The various physical resources consumed in physical layer to this module are monitored and monitoring information are fed back
Give platform O＆M.
Operation audit：Critical operations are carried out with the record of audit information, inquiry is provided platform O＆M and led in real time
The function such as know.
Optionally, the MDAC, is used for：
Create connection and send SQL request；
Wherein, connection, including correct configuration JDBC parameters are created, JDBC parameters are transmitted to connection management module, connecting tube
Reason template provides parameter to state server according to JDBC versions and verified, obtains metadata information；
SQL request, including whether the SQL connections for judging to generate are to penetrate connection, if connection is penetrated, then based on given
Physical database create or select on demand connection.
The invention provides a kind of inquiry system for multi-source data, including service access application programming interfaces, it is used for
The data access request from service application is received, access request is forwarded to MDAC, MDAC extraction
Transfer Parameters are judged by the Transfer Parameters in access request；JDBC HUB engine container modules, for when judged result table
When bright data access is inter-library request of data, makes MDAC that data access request is forwarded into distributed data inquiry and draw
Hold up；Application database drives layer, for making distributed data query engine be connected to more data corresponding with data access request
Source, Query Result corresponding with data access request is obtained at multi-data source；Service access application programming interfaces, be additionally operable to by
Query Result is forwarded to MDAC so that data forwarding component sends Query Result to service application.Pass through foundation
One database in logic, isolate between service application and database and be directly connected to, so as to improve the security of data access；
Abstract exploitation service is provided, simplifies the exploitation of data management and application；The correlation inquiry between physically inter-library table is supported,
Realize the multi-source data inquiry of integration across database.
Each sequence number in above-described embodiment is for illustration only, does not represent the elder generation during the assembling or use of each part
Order afterwards.
Embodiments of the invention are the foregoing is only, are not intended to limit the invention, it is all in the spirit and principles in the present invention
Within, any modification, equivalent substitution and improvements made etc., it should be included in the scope of the protection.
Claims (10)
1. a kind of querying method for multi-source data, the querying method is used to provide between service application layer and database
Data access response, it is characterised in that the querying method, including：
The data access request from service application is received, access request is forwarded to MDAC, MDAC
The Transfer Parameters in access request are extracted, Transfer Parameters are judged；
When it is inter-library request of data that judged result, which shows data access, MDAC is made to be forwarded to data access request
Distributed data query engine；
Make distributed data query engine be connected to multi-data source corresponding with data access request, at multi-data source obtain with
Query Result corresponding to data access request, Query Result is forwarded to MDAC so that data forwarding component will be looked into
Result is ask to send to service application.
A kind of 2. querying method and system for multi-source data according to claim 1, it is characterised in that the data
The form of expression of access request is SQL.
A kind of 3. querying method and system for multi-source data according to claim 2, it is characterised in that the data
Access component is offer standard java Java DataBase Connections, the interface docked with service application.
4. the inquiry system for multi-source data according to any one of claims 1 to 3, it is characterised in that the data
Component is accessed, is used for：
Create connection and send SQL request；
Wherein, connection, including correct configuration JDBC parameters are created, JDBC parameters are transmitted to connection management module, connection management mould
Plate provides parameter to state server according to JDBC versions and verified, obtains metadata information；
SQL request, including whether the SQL connections for judging to generate are to penetrate connection, if connection is penetrated, then based on given thing
Reason database creates or selected on demand connection.
5. a kind of inquiry system for multi-source data, the querying method is used to provide between service application layer and database
Data access response, it is characterised in that the inquiry system, including：
Service access application programming interfaces, for receiving the data access request from service application, access request is forwarded to
MDAC, MDAC extract the Transfer Parameters in access request, Transfer Parameters are judged；
JDBC HUB engine container modules, for when it is inter-library request of data that judged result, which shows data access, making data visit
Ask that data access request is forwarded to distributed data query engine by component；
Application database drives layer, for making distributed data query engine be connected to more data corresponding with data access request
Source, Query Result corresponding with data access request is obtained at multi-data source；
Service access application programming interfaces, it is additionally operable to Query Result being forwarded to MDAC so that data forwarding component
Query Result is sent to service application.
6. the inquiry system according to claim 5 for multi-source data, it is characterised in that the service access application journey
Sequence interface, is specifically used for：
JDBC agreement access interfaces are provided, the command request that application database driving is sent are captured, to operation system and business
Database transparent unaware, support multiple database hybrid protocol；
The parameter transmitted by SQL connections judges need which kind of types of database accessed, and is parsed, analyzed and processed.
7. the inquiry system according to claim 5 for multi-source data, it is characterised in that the JDBC HUB engines hold
Device module, including：
SQL parsings, data route, driven management, data source control, performance evaluation, operation audit, data parsing and processing, SQL
Performance data collection, rights management and storehouse table management；
Wherein, rights management includes carrying out storehouse, the other rights management of table level and control, and fine-grained authority relies on bottom data source
To control；
Driven management includes the various activation bits of user's registration, supports the management of multi version driving；
Instance management includes the various systematic parameters for being supplied to platform operation maintenance personnel configuration management JDBC HUB examples.
8. the inquiry system according to claim 7 for multi-source data, it is characterised in that the SQL parsings, including：
The request analysis of application is distributed to specific data source for SQL and prepared to relevant informations such as storehouse, tables；Final request will
Each relevant database is connected to by the adapter of relevant database.
9. the inquiry system according to claim 7 for multi-source data, it is characterised in that the data route, including：
Data are route：According to the SQL results parsed and the information of storehouse table name word space management, it is determined that being adapted to using which data source
Device and final data source, and establish connection and request is forwarded, for the raising of performance, support connection pool etc. and optimization skill
Art, while be the access point of various processing plug-in units, such as performance collection module.
10. the inquiry system for multi-source data according to any one of claim 5 to 9, it is characterised in that the data
Component is accessed, is used for：
Create connection and send SQL request；
Wherein, connection, including correct configuration JDBC parameters are created, JDBC parameters are transmitted to connection management module, connection management mould
Plate provides parameter to state server according to JDBC versions and verified, obtains metadata information；
SQL request, including whether the SQL connections for judging to generate are to penetrate connection, if connection is penetrated, then based on given thing
Reason database creates or selected on demand connection.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710808772.1A CN107818127A (en) | 2017-09-09 | 2017-09-09 | A kind of querying method and system for multi-source data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710808772.1A CN107818127A (en) | 2017-09-09 | 2017-09-09 | A kind of querying method and system for multi-source data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107818127A true CN107818127A (en) | 2018-03-20 |
Family
ID=61601602
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710808772.1A Pending CN107818127A (en) | 2017-09-09 | 2017-09-09 | A kind of querying method and system for multi-source data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107818127A (en) |
Cited By (35)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108932345A (en) * | 2018-07-27 | 2018-12-04 | 北京中关村科金技术有限公司 | One kind realizing across data source distributed Query Processing System and method based on dremio |
| CN109145025A (en) * | 2018-09-14 | 2019-01-04 | 阿里巴巴集团控股有限公司 | A kind of data query method, apparatus and service server that multi-data source is integrated |
| CN109285068A (en) * | 2018-10-25 | 2019-01-29 | 平安普惠企业管理有限公司 | Online loan inquiry method, apparatus, equipment and storage medium |
| CN109325069A (en) * | 2018-09-07 | 2019-02-12 | 腾讯科技（深圳）有限公司 | Method for processing business, device and the network equipment |
| CN109446179A (en) * | 2018-09-30 | 2019-03-08 | 天津天地伟业电子工业制造有限公司 | A method of it is concurrent that SQLITE database realizing multi-service being established based on Memory mode |
| CN109446253A (en) * | 2018-09-25 | 2019-03-08 | 平安科技（深圳）有限公司 | Data query control method, device, computer equipment and storage medium |
| CN109508344A (en) * | 2018-09-29 | 2019-03-22 | 中国平安人寿保险股份有限公司 | Business datum querying method, device, electronic equipment and storage medium |
| CN109582717A (en) * | 2018-10-16 | 2019-04-05 | 中国电力科学研究院有限公司 | A kind of database unified platform and its read method towards electric power big data |
| CN109657128A (en) * | 2018-12-20 | 2019-04-19 | 北京小米移动软件有限公司 | Data query method, apparatus and storage medium |
| CN109656968A (en) * | 2018-11-15 | 2019-04-19 | 中国建设银行股份有限公司 | Data query method, apparatus and storage medium under distributed environment |
| CN109656989A (en) * | 2018-10-29 | 2019-04-19 | 平安科技（深圳）有限公司 | Multi-data source integration method, device, computer equipment and storage medium |
| CN109947789A (en) * | 2019-01-28 | 2019-06-28 | 平安科技（深圳）有限公司 | A kind of method, apparatus, computer equipment and the storage medium of the data processing of multiple database |
| CN109992617A (en) * | 2019-03-21 | 2019-07-09 | 国电南瑞科技股份有限公司 | Support the data access method and system of a variety of programming languages and multiple data sources |
| CN110209719A (en) * | 2019-05-20 | 2019-09-06 | 华南理工大学 | A kind of multitype database based on micro services framework uniformly accesses system and method |
| CN110321375A (en) * | 2019-03-11 | 2019-10-11 | 浙江中控软件技术有限公司 | A kind of data communication system and method that multi-data source is integrated |
| CN110489442A (en) * | 2019-08-22 | 2019-11-22 | 行吟信息科技（上海）有限公司 | A kind of data query method and system |
| CN110543498A (en) * | 2019-08-20 | 2019-12-06 | 武汉稀云科技有限公司 | Multi-party data association query method and device based on event triggering |
| CN110909059A (en) * | 2019-11-25 | 2020-03-24 | 杭州晨鹰军泰科技有限公司 | Data integration system, method, equipment and storage medium |
| CN110928712A (en) * | 2019-11-28 | 2020-03-27 | 广州华资软件技术有限公司 | Method for forming standard by connecting RPC service interface through visual database |
| CN110955712A (en) * | 2019-11-26 | 2020-04-03 | 北京天元创新科技有限公司 | Development API processing method and device based on multiple data sources |
| CN111176857A (en) * | 2018-11-12 | 2020-05-19 | 北京数安鑫云信息技术有限公司 | Method and device for intelligently providing data |
| CN111756752A (en) * | 2020-06-24 | 2020-10-09 | 北京金山云网络技术有限公司 | Method and device for controlling access authority of database and electronic equipment |
| CN112328704A (en) * | 2020-11-03 | 2021-02-05 | 成都中科大旗软件股份有限公司 | Method, system, computer equipment and storage medium for realizing multi-data source combined query |
| CN112351099A (en) * | 2020-11-06 | 2021-02-09 | 北京金山云网络技术有限公司 | Data access method and device and server |
| CN112685443A (en) * | 2021-01-12 | 2021-04-20 | 树根互联技术有限公司 | Data query method and device, electronic equipment and computer readable storage medium |
| CN112703514A (en) * | 2018-07-17 | 2021-04-23 | 优胜视域股份有限公司 | Business operating system engine |
| CN113315707A (en) * | 2021-05-26 | 2021-08-27 | 西安热工研究院有限公司 | Method for label routing of multi-site cascade mode data transmission middleware |
| CN113609141A (en) * | 2021-10-11 | 2021-11-05 | 武大吉奥信息技术有限公司 | Non-invasive cross-database data fusion method based on API splicing |
| CN114329107A (en) * | 2021-12-31 | 2022-04-12 | 浙江力石科技股份有限公司 | Multi-data-source joint query method based on global data dictionary |
| CN114756577A (en) * | 2022-03-25 | 2022-07-15 | 北京友友天宇***技术有限公司 | Processing method of multi-source heterogeneous data, computer equipment and storage medium |
| CN114840608A (en) * | 2022-07-04 | 2022-08-02 | 深圳市茗格科技有限公司 | Distributed data access method, device and storage medium |
| CN115269561A (en) * | 2022-09-21 | 2022-11-01 | 国网智能电网研究院有限公司 | Hybrid database management method and device, hybrid database and electronic equipment |
| CN115587123A (en) * | 2022-12-07 | 2023-01-10 | 北京安博通科技股份有限公司 | Method, device, equipment and storage medium for searching different databases based on AQL |
| CN116126944A (en) * | 2023-02-14 | 2023-05-16 | 中电金信软件有限公司 | Data access system, method and electronic equipment |
| CN117033748A (en) * | 2023-10-08 | 2023-11-10 | 恒生电子股份有限公司 | Service query information acquisition method, service data query method and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101013986A (en) * | 2007-02-02 | 2007-08-08 | 南京邮电大学 | Method for realizing data inquiring system of sensor network based on middleware of mobile agent |
| CN101739460A (en) * | 2009-12-16 | 2010-06-16 | 中国科学院对地观测与数字地球科学中心 | Grid-based spatial data source unification service system and method |
| CN103440303A (en) * | 2013-08-21 | 2013-12-11 | 曙光信息产业股份有限公司 | Heterogeneous cloud storage system and data processing method thereof |
| CN104008135A (en) * | 2014-05-07 | 2014-08-27 | 南京邮电大学 | Multi-source heterogeneous database fusion system and data query method thereof |
| CN106874461A (en) * | 2017-02-14 | 2017-06-20 | 北京慧正通软科技有限公司 | A kind of workflow engine supports multi-data source configuration security access system and method |
| CN107066499A (en) * | 2016-12-30 | 2017-08-18 | 江苏瑞中数据股份有限公司 | The data query method of multi-source data management and visualization system is stored towards isomery |
-
2017
- 2017-09-09 CN CN201710808772.1A patent/CN107818127A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101013986A (en) * | 2007-02-02 | 2007-08-08 | 南京邮电大学 | Method for realizing data inquiring system of sensor network based on middleware of mobile agent |
| CN101739460A (en) * | 2009-12-16 | 2010-06-16 | 中国科学院对地观测与数字地球科学中心 | Grid-based spatial data source unification service system and method |
| CN103440303A (en) * | 2013-08-21 | 2013-12-11 | 曙光信息产业股份有限公司 | Heterogeneous cloud storage system and data processing method thereof |
| CN104008135A (en) * | 2014-05-07 | 2014-08-27 | 南京邮电大学 | Multi-source heterogeneous database fusion system and data query method thereof |
| CN107066499A (en) * | 2016-12-30 | 2017-08-18 | 江苏瑞中数据股份有限公司 | The data query method of multi-source data management and visualization system is stored towards isomery |
| CN106874461A (en) * | 2017-02-14 | 2017-06-20 | 北京慧正通软科技有限公司 | A kind of workflow engine supports multi-data source configuration security access system and method |
Cited By (51)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112703514A (en) * | 2018-07-17 | 2021-04-23 | 优胜视域股份有限公司 | Business operating system engine |
| CN108932345A (en) * | 2018-07-27 | 2018-12-04 | 北京中关村科金技术有限公司 | One kind realizing across data source distributed Query Processing System and method based on dremio |
| CN109325069A (en) * | 2018-09-07 | 2019-02-12 | 腾讯科技（深圳）有限公司 | Method for processing business, device and the network equipment |
| CN109145025A (en) * | 2018-09-14 | 2019-01-04 | 阿里巴巴集团控股有限公司 | A kind of data query method, apparatus and service server that multi-data source is integrated |
| CN109145025B (en) * | 2018-09-14 | 2021-09-24 | 创新先进技术有限公司 | Multi-data-source integrated data query method and device and service server |
| CN109446253A (en) * | 2018-09-25 | 2019-03-08 | 平安科技（深圳）有限公司 | Data query control method, device, computer equipment and storage medium |
| CN109446253B (en) * | 2018-09-25 | 2024-05-07 | 平安科技（深圳）有限公司 | Data query control method, device, computer equipment and storage medium |
| CN109508344B (en) * | 2018-09-29 | 2024-03-12 | 中国平安人寿保险股份有限公司 | Service data query method and device, electronic equipment and storage medium |
| CN109508344A (en) * | 2018-09-29 | 2019-03-22 | 中国平安人寿保险股份有限公司 | Business datum querying method, device, electronic equipment and storage medium |
| CN109446179A (en) * | 2018-09-30 | 2019-03-08 | 天津天地伟业电子工业制造有限公司 | A method of it is concurrent that SQLITE database realizing multi-service being established based on Memory mode |
| CN109582717B (en) * | 2018-10-16 | 2022-06-07 | 中国电力科学研究院有限公司 | Database unified platform for electric power big data and reading method thereof |
| CN109582717A (en) * | 2018-10-16 | 2019-04-05 | 中国电力科学研究院有限公司 | A kind of database unified platform and its read method towards electric power big data |
| CN109285068A (en) * | 2018-10-25 | 2019-01-29 | 平安普惠企业管理有限公司 | Online loan inquiry method, apparatus, equipment and storage medium |
| CN109656989A (en) * | 2018-10-29 | 2019-04-19 | 平安科技（深圳）有限公司 | Multi-data source integration method, device, computer equipment and storage medium |
| CN111176857A (en) * | 2018-11-12 | 2020-05-19 | 北京数安鑫云信息技术有限公司 | Method and device for intelligently providing data |
| CN111176857B (en) * | 2018-11-12 | 2024-06-11 | 北京数安鑫云信息技术有限公司 | Method and device for intelligently providing data |
| CN109656968A (en) * | 2018-11-15 | 2019-04-19 | 中国建设银行股份有限公司 | Data query method, apparatus and storage medium under distributed environment |
| CN109657128B (en) * | 2018-12-20 | 2021-02-26 | 北京小米移动软件有限公司 | Data query method, device and storage medium |
| CN109657128A (en) * | 2018-12-20 | 2019-04-19 | 北京小米移动软件有限公司 | Data query method, apparatus and storage medium |
| CN109947789B (en) * | 2019-01-28 | 2023-12-19 | 平安科技（深圳）有限公司 | Method, device, computer equipment and storage medium for processing data of multiple databases |
| CN109947789A (en) * | 2019-01-28 | 2019-06-28 | 平安科技（深圳）有限公司 | A kind of method, apparatus, computer equipment and the storage medium of the data processing of multiple database |
| CN110321375A (en) * | 2019-03-11 | 2019-10-11 | 浙江中控软件技术有限公司 | A kind of data communication system and method that multi-data source is integrated |
| CN109992617A (en) * | 2019-03-21 | 2019-07-09 | 国电南瑞科技股份有限公司 | Support the data access method and system of a variety of programming languages and multiple data sources |
| CN109992617B (en) * | 2019-03-21 | 2022-08-26 | 国电南瑞科技股份有限公司 | Data access method and system supporting multiple programming languages and multiple data sources |
| CN110209719A (en) * | 2019-05-20 | 2019-09-06 | 华南理工大学 | A kind of multitype database based on micro services framework uniformly accesses system and method |
| CN110543498B (en) * | 2019-08-20 | 2022-02-18 | 武汉稀云科技有限公司 | Multi-party data association query method and device based on event triggering |
| CN110543498A (en) * | 2019-08-20 | 2019-12-06 | 武汉稀云科技有限公司 | Multi-party data association query method and device based on event triggering |
| CN110489442A (en) * | 2019-08-22 | 2019-11-22 | 行吟信息科技（上海）有限公司 | A kind of data query method and system |
| CN110909059A (en) * | 2019-11-25 | 2020-03-24 | 杭州晨鹰军泰科技有限公司 | Data integration system, method, equipment and storage medium |
| CN110955712A (en) * | 2019-11-26 | 2020-04-03 | 北京天元创新科技有限公司 | Development API processing method and device based on multiple data sources |
| CN110928712A (en) * | 2019-11-28 | 2020-03-27 | 广州华资软件技术有限公司 | Method for forming standard by connecting RPC service interface through visual database |
| CN110928712B (en) * | 2019-11-28 | 2023-03-28 | 广州华资软件技术有限公司 | Method for forming standard by connecting RPC service interface through visual database |
| CN111756752A (en) * | 2020-06-24 | 2020-10-09 | 北京金山云网络技术有限公司 | Method and device for controlling access authority of database and electronic equipment |
| CN112328704A (en) * | 2020-11-03 | 2021-02-05 | 成都中科大旗软件股份有限公司 | Method, system, computer equipment and storage medium for realizing multi-data source combined query |
| CN112328704B (en) * | 2020-11-03 | 2024-02-23 | 成都中科大旗软件股份有限公司 | Method, system, computer device and storage medium for implementing multiple data source joint inquiry |
| CN112351099A (en) * | 2020-11-06 | 2021-02-09 | 北京金山云网络技术有限公司 | Data access method and device and server |
| CN112351099B (en) * | 2020-11-06 | 2023-03-24 | 北京金山云网络技术有限公司 | Data access method and device and server |
| CN112685443B (en) * | 2021-01-12 | 2023-11-10 | 树根互联股份有限公司 | Data query method, device, electronic equipment and computer readable storage medium |
| CN112685443A (en) * | 2021-01-12 | 2021-04-20 | 树根互联技术有限公司 | Data query method and device, electronic equipment and computer readable storage medium |
| CN113315707A (en) * | 2021-05-26 | 2021-08-27 | 西安热工研究院有限公司 | Method for label routing of multi-site cascade mode data transmission middleware |
| CN113609141A (en) * | 2021-10-11 | 2021-11-05 | 武大吉奥信息技术有限公司 | Non-invasive cross-database data fusion method based on API splicing |
| CN114329107A (en) * | 2021-12-31 | 2022-04-12 | 浙江力石科技股份有限公司 | Multi-data-source joint query method based on global data dictionary |
| CN114756577A (en) * | 2022-03-25 | 2022-07-15 | 北京友友天宇***技术有限公司 | Processing method of multi-source heterogeneous data, computer equipment and storage medium |
| CN114840608A (en) * | 2022-07-04 | 2022-08-02 | 深圳市茗格科技有限公司 | Distributed data access method, device and storage medium |
| CN115269561A (en) * | 2022-09-21 | 2022-11-01 | 国网智能电网研究院有限公司 | Hybrid database management method and device, hybrid database and electronic equipment |
| CN115269561B (en) * | 2022-09-21 | 2023-01-24 | 国网智能电网研究院有限公司 | Hybrid database management method and device, hybrid database and electronic equipment |
| CN115587123B (en) * | 2022-12-07 | 2023-09-26 | 北京安博通科技股份有限公司 | Method, device, equipment and storage medium for searching different databases based on AQL |
| CN115587123A (en) * | 2022-12-07 | 2023-01-10 | 北京安博通科技股份有限公司 | Method, device, equipment and storage medium for searching different databases based on AQL |
| CN116126944A (en) * | 2023-02-14 | 2023-05-16 | 中电金信软件有限公司 | Data access system, method and electronic equipment |
| CN117033748A (en) * | 2023-10-08 | 2023-11-10 | 恒生电子股份有限公司 | Service query information acquisition method, service data query method and device |
| CN117033748B (en) * | 2023-10-08 | 2024-02-06 | 恒生电子股份有限公司 | Service query information acquisition method, service data query method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107818127A (en) | A kind of querying method and system for multi-source data | |
| CN103530354B (en) | geographic information data fusion system | |
| CN107193546B (en) | Micro-service business application system | |
| CN108964996B (en) | Urban and rural integrated information grid system and information sharing method based on same | |
| CN107203376B (en) | Enterprise-level informatization system | |
| CN109144982A (en) | Multidimensional holographic Database Dynamic constructing technology system | |
| CN112688865B (en) | Design method of OPC UA gateway for graphical online modeling | |
| CN105740411B (en) | A kind of data migration method based on SOA and WebService | |
| CN103136249A (en) | System and method of multiplex mode isomerous data integration | |
| CN106470216A (en) | A kind of Content Management System based on information sharing, interaction | |
| CN105574698A (en) | Intelligent storage management system based on big data | |
| CN109542967A (en) | Smart city data-sharing systems and method based on XBRL standard | |
| CN109582717A (en) | A kind of database unified platform and its read method towards electric power big data | |
| CN112698953A (en) | Power grid intelligent operation and detection platform based on micro-service | |
| CN107122415B (en) | The integrated system of place name address library data fusion | |
| CN102882896A (en) | Data information access system and method based on internet of things | |
| CN105787823A (en) | Scientific and technological innovation linkage information service platform and realization method thereof | |
| CN111582773A (en) | Multi-tenant technology-based micro-grid energy cloud model control method and system | |
| CN113778983A (en) | Internet of things equipment and data standardized access and summary presentation method | |
| CN109523423B (en) | Application system generation method, device, equipment and storage medium | |
| CN106357518A (en) | Material warehousing cabinet as well as material warehousing managing method, server and system | |
| CN111652374B (en) | Smart city perception equipment resource management method and system | |
| CN106874114A (en) | Express delivery management software system | |
| CN107656951A (en) | A kind of method of real time data in synchronous and heterogeneous Database Systems | |
| CN110888878A (en) | Service-oriented main data management method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180320 |