CN104915341A - Visual multi-database ETL integration method and system - Google Patents
Visual multi-database ETL integration method and system Download PDFInfo
- Publication number
- CN104915341A CN104915341A CN201410086142.4A CN201410086142A CN104915341A CN 104915341 A CN104915341 A CN 104915341A CN 201410086142 A CN201410086142 A CN 201410086142A CN 104915341 A CN104915341 A CN 104915341A
- Authority
- CN
- China
- Prior art keywords
- database
- etl
- source
- data
- sql statement
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Abstract
The invention provides a visual multi-database ETL integration method and system. The visual multi-database ETL integration method includes the following steps that source databases and target databases are connected; the SQL statements of source lists of the source databases are obtained through ETL matching of the source databases and the target databases; the SQL statements are optimized and executed, and the ETL data of the multiple source databases are obtained and introduced into target lists of the target databases. The visual multi-database ETL integration system comprises a database management system layer and a semantic layer, the database management system layer connects the source databases with the target databases, and the SQL statements of the source lists of the source databases are obtained through ETL matching of the source databases and the target databases; the semantic layer optimizes and executes the SQL statements, obtains the ETL data of the multiple source databases and introduces the ETL data to the target lists of the target databases. The visual multi-database ETL integration method and system reduce the complexity degree of integration of the multiple databases, improve the integration efficiency of the multiple databases, and reduce the risk of integration of the multiple databases.
Description
Technical field
The present invention relates to geo-database integration exploitation and database running optimizatin field, is that a set of by graphic interface configuration, to realize multitype database integrated, the integrated approach of data pick-up and injection and system.
Background technology
Along with the develop rapidly of infotech, database application is more and more extensive, because department service is different with function ownership, different database environments is have employed during each application system development, very large difficulty is brought to practical application, the Integrated predict model technology of multiple database is a difficult problem always, and majority includes software system integration application, the visualization problem of database, the analytical algorithm etc. of data according to application technology.At present, although exist about geo-database integration method, be on the one hand that the integration degree of these class methods is not high, it is low to carry out Query Efficiency after integrated; Be the very flexible of these class methods in addition, configure more loaded down with trivial details, if when running into the tables of data of design and field more complicated and various situation, not only need a large amount of time configurations, and easily make mistakes.
At present, the main difficulty of existence is integrated, the efficiency that data migration process optimization problem, disparate databases are integrated of multiple database.Because framework is different, multiple database is integrated needs multiple technologies support; Data migration process affects by data magnitude, and a large amount of Data Migrations can cause database operational efficiency to reduce, and affects the use of database; The diversity of client can affect integrated efficiency, and between client and client, client and database side exist channel transmission, and transfer efficiency can be caused low.
Summary of the invention
The object of this invention is to provide a set of visual many ETL process integrated approach.The technical scheme that the present invention is adopted for achieving the above object is:
Visual multiple database ETL integrated approach, comprises the following steps:
Connect source database and target database; Mated by the ETL of source database and target database, obtain the SQL statement of the source table of source database;
SQL statement is optimized and performs, obtain the ETL data of multiple source database and be injected into the object table of target database.
The described ETL by source database and target database mates, and the SQL statement obtaining the source table of source database comprises the following steps:
Source of configuration database and target database and table name, field, and judge the type of database of source database and target database;
Type of database according to source database and target database determines different ETL Regularias, then shows according to the source of source database the SQL statement being obtained source table by ETL Regularia;
Describedly SQL statement is optimized and performs, obtains the ETL data of multiple source database and the object table being injected into target database comprises the following steps:
Treatment S QL statement is also optimized SQL statement according to the result of ETL rule base coupling;
The API that calling platform layer provides, the SQL statement performed after optimizing obtains ETL data and stored in data buffer, according to ETL rule base, ETL data is injected the object table of object library.
Described treatment S QL statement being optimized SQL statement according to the result of ETL rule base coupling comprises the following steps:
SQL statement is set up a tree construction; Semantic test is carried out to each node of tree construction, and carries out cooperating measure, parsing tree is converted to the algebraic manipulation symbol tree of the inquiry plan representing initial; Algebraic manipulation symbol tree is converted to the fastest SQL sequence of execution speed.
Visual multiple database ETL integrated system, comprising:
Data base management system (DBMS) layer: connect source database and target database; Mated by the ETL of source database and target database, obtain the SQL statement of the source table of source database;
Semantic layer: SQL statement is optimized and performs, obtain the ETL data of multiple source database and be injected into the object table of target database.
Described data base management system (DBMS) layer comprises:
Graphical configuration interface: source of configuration database and target database and table name, field, and judge the type of database of source database and target database;
ETL rule base: the type of database according to source database and target database determines different ETL Regularias, then obtain SQL statement according to the source table of source database by ETL Regularia;
Described semantic layer comprises:
Query compiler device: treatment S QL statement is also optimized SQL statement according to the result of ETL rule base coupling;
Enforcement engine: the API that calling platform layer provides, the SQL statement performed after optimizing obtains ETL data and stored in data buffer, according to ETL rule base, ETL data is injected the object table of object library.
Described query compiler device comprises:
Query analyzer: SQL statement is set up a tree construction;
Inquiry pretreater: carry out semantic test to each node of tree construction, and carry out semantic test to each node of tree construction, and carry out cooperating measure, is converted to the algebraic manipulation symbol tree of the inquiry plan representing initial by parsing tree;
Query optimizer: algebraic manipulation symbol tree is converted to the fastest SOL sequence of execution speed.
The present invention has following beneficial effect and advantage:
1. the present invention reduces the integrated complexity of multiple database, improves the efficiency of geo-database integration, reduces the risk of geo-database integration.
2. the present invention is designed by hierarchical logic, solve the data source of each database at hardware platform, the difference problem of operating system and communication protocol, a higher abstraction hierarchy carries out the design of applied logic, decrease the complicacy that system realizes, and make system have good exploration and extendability.
3. the present invention is by query compiler device, query optimizer and enforcement engine, completes data query and unloading fast.
4. the present invention is by graphical configuration interface and ETL rule base, realizes carrying out grammatical analysis and verification, to avoid mistake, ensures the correct execution of data pick-up.
5. the present invention is by podium level, and achieve the support of system to data source, the system that ensure that is integrated flexibly in different platform.
Accompanying drawing explanation
Fig. 1 is visual multiple database ETL integrated approach frame diagram of the present invention;
Fig. 2 is ETL process flow diagram flow chart.
Embodiment
Below in conjunction with drawings and Examples, the present invention is described in further detail.
ETL (Extract, Transform, Load) comprises the extraction of data, conversion and loading.This method utilizes patterned human-computer interaction interface to configure multiple data source, then from these source databases, data are obtained according to certain ETL logical method, after conversion, be loaded into target data to process, in order to improve the operational efficiency of database in whole process, ETL logical method can be optimized decomposition according to the feature of distinct type data-base sql like language.
The present invention, by graphical human-computer interaction interface, completes the configuration of multi-data source, by configuration ETL rule, obtains data, after conversion, be loaded into target database and process, complete data integration from source database.Concrete steps are as follows:
1) define data source, carry out database by graphical configuration interface, table name, the configuration of field.
2) analyze data source characteristic, propose unified configuration interface, define the adapter for Oracle, DB2, SQLSERVER, SYBASE.
3) define query compiler device, by the inquiry of textual form, set up a tree construction, semantic test is carried out to inquiry, form the initial sequence of operation.
4) define query optimizer, utilize statistics to determine the fastest sequence of operation.
5) define enforcement engine, each step in the sequence of operation that responsible execution is chosen, operates, and be put in buffer zone, and scheduler carries out alternately to data, to avoid accessing the data having been added lock.
6) define ETL rule, read the data of buffer zone, and carry out unloading, check data by graphical interfaces.
Described semantic layer, definition query compiler device, query optimizer and enforcement engine, by text, forming the efficient sequence of operation by optimizing, being responsible for database mutual by enforcement engine.
The design of visual multiple database ETL integrated approach is mainly reflected in three levels: podium level, semantic layer and data base management system (DBMS) layer, as Fig. 1.Designed by hierarchical logic, solve the data source of each database at hardware platform, the difference problem of operating system and communication protocol, a higher abstraction hierarchy carries out the design of applied logic, decrease the complicacy that system realizes, and make system have good exploration and extendability.
1) podium level:
Comprise the various application needed by multidatabase system layer external interface Access Integration information.Podium level comprises the configuration interface of hardware information, operating system and communication protocol, for other application layers, the interface that accessing database calling platform layer provides, just looks like the same at access database, reaches the effect of simultaneously accessing data in multiple database.
2) semantic layer:
Comprise two parts: query compiler device and enforcement engine.
A) query compiler device:
Query translation is become a kind of internal form by query compiler device, is called inquiry plan.Inquiry plan is the sequence of operations that will perform in data.Usually, the operation in inquiry plan is the realization of " relational algebra ".
Query compiler device comprises:
1. query analyzer, it is by the inquiry of textual form, sets up a tree construction.
2. inquire about pretreater, it carries out semantic test to inquiry and (such as, checks that the relation mentioned in inquiry is
No existence all really), and carry out some tree construction conversion, parsing tree is converted to the algebraic manipulation symbol tree of the inquiry plan representing initial.
3. query optimizer, initial inquiry plan is converted to the most effective sequence of operation for real data by it.Query compiler device utilize metadata and about the statistics of data to determine which sequence of operation may be the fastest.
B) enforcement engine:
Enforcement engine is responsible for performing each step in the inquiry plan chosen.In order to operate data, enforcement engine must be put into database data in buffer zone, and scheduler carries out alternately, to avoid accessing the data having been added lock.In whole process, any event will by the corresponding log information of log manager record.
3) data base management system (DBMS) layer:
Comprise graphical configuration interface and ETL rule base.Carrying out in multitype database operating process, for ensureing the consistance of table name in rule, field name, field type, meeting the constraint condition of data, through the decimation rule that visual edit is set up, need to carry out grammatical analysis and verification, to avoid mistake, ensure the correct execution of data pick-up.
The execution of ETL process is divided into 4 parts: be 1. connected to data source; 2. resolve SQL statement according to decimation rule, and utilize the SQL statement performed after principle of optimality optimization is resolved, from one or more tables of data of source database, inquiry obtains intermediate result data; 3. conversion process intermediate result data, obtains result data; 4. result data loads (storage) in the object table of target database.As can be seen here, ETL process contains the complete information performed needed for data pick-up work, comprises source database, source data table, extraction and performs the principle of optimality, transformation rule, target database, target matrix etc.
Each level of the present invention realizes different functions, below for specifically describing:
1) podium level
Comprise the various application needed by multidatabase system layer external interface Access Integration information.Podium level comprises the configuration interface of hardware information, operating system, database and communication protocol, hardware platform mainly contains minicomputer, microcomputer server, operating system has Windows, Unix, AIX etc., database has Oracle, DB2, SQLSERVER, SYBASE, by unified configuration interface, realize the Seamless integration-of application system, for other application layers, the interface that accessing database calling platform layer provides, just look like the same at access database, reach the effect of simultaneously accessing data in multiple database.
2) semantic layer
Semantic layer realizes comprising the structure of query compiler device and enforcement engine, and different data source query statements is different, according to data source characteristic, determines semanteme, and carries out resolving, optimizes, performs.
3) data base management system (DBMS) layer
Utilize the graphic interface provided to carry out data, the configuration of tables of data and field, comprise database and tables of data configuration, service configuration mainly comprises address of service and upgrades and DataBase combining renewal.
4) ETL rule base
First different according to data source, determine different ETL Regularias, secondly for concrete data source, formulate ETL rule, the structure in implementation rule storehouse.Rule base is used to define and store data pick-up, data conversion and Data import rule.
As shown in Figure 2:
STEP1: obtain and resolve source database connection string; Data source comprises EXCEL data source, flat file data source, relational database data source etc., and wherein relational database data source comprises Oracle, Sql Server, DB2 etc.Data source connection string needs to confirm following field, for Oracle, comprising: server name, database-name, user, password etc.
STEP2: obtain and resolve target database connection string; Target database kind is identical with data source kind, and connection string field information is substantially identical.Hypothetical target database is Sql Server.
STEP3: carry out database by graphical configuration interface, table name, the configuration of field; For individual data source:
Data source (Oracle) source table data dictionary is as shown in table 1, and source table is as shown in table 2:
Table 1
| Field name | Field type | Major key |
| EquipID | VARchar2（32） | Y |
| EquipName | VARchar2（64） | |
| Sub_Equip_Flag | VARchar2（1） | |
| Up_Down_Flag | VARchar2（1） |
Table 2
| EquipID | EquipName | Sub_Equip_Flag | Up_Down_Flag |
| DA_001 | 1# equipment | N | U |
| DA_001A | 1# subset | Y | U |
| DA_002 | 2# equipment | N | D |
Target database (Sql Server) object table data dictionary is as shown in table 3:
Table 3
| Field name | Field type | Major key |
| EquipID | NVARchar（32） | Y |
| EquipName | NVARchar（64） | |
| Sub_Flag | BIT | |
| Equip_Status | BIT |
Define field matched rule is as shown in table 4:
Table 4
| Oracle(source) | Sql Server(target) |
| EquipID | EquipID |
| EquipName | EquipName |
| Sub_Equip_Flag | Sub_Flag |
| Up_Down_Flag | Equip_Status |
Data type conversion is analyzed as shown in table 5:
Table 5
Definition value transformation rule is as shown in table 6:
Table 6
Definition data type conversion rule is as shown in table 7:
Table 7
Definition Data Division rule:
1) loaded targets database.
2) definition splits rule: in target database, there is record, be expressed as Condition1; For there is record in target database, be expressed as Condition2.
3) Condition1 is defined, rules process method.
4) Conditon2 is defined, rules process method.
STEP4: the type of database judging source database and object library, and carry out ETL rule base coupling;
STEP5: connection data source and target database;
STEP6: resolve ETL decimation rule character string, obtain the SQL statement of data source table;
STEP7: start query compiler device, treatment S QL statement also verifies whether there is conflict and mistake;
STEP8: starting guide engine (i.e. query compiler device), the result according to ETL rule base coupling is optimized SQL statement;
STEP9: start enforcement engine, the API that calling platform layer provides, performs the SQL statement after optimizing;
STEP10: obtain multiple database ETL data, stored in data buffer;
STEP11: according to ETL regulation engine, disposal data buffer zone stored in data, and be injected into object table;
STEP12: whether checking injects effective, and points out.
The present invention has been successfully applied in the integration of information system of plurality of classes, and such as manufacturing execution system, financial system, centralized control system etc., by application of the present invention, also drastically increase the integrated convenience of multiple database and stability.
Claims (8)
1. visual multiple database ETL integrated approach, is characterized in that comprising the following steps:
Connect source database and target database; Mated by the ETL of source database and target database, obtain the SQL statement of the source table of source database;
SQL statement is optimized and performs, obtain the ETL data of multiple source database and be injected into the object table of target database.
2. visual multiple database ETL integrated approach according to claim 1, is characterized in that, the described ETL by source database and target database mates, and the SQL statement obtaining the source table of source database comprises the following steps:
Source of configuration database and target database and table name, field, and judge the type of database of source database and target database;
Type of database according to source database and target database determines different ETL Regularias, then shows according to the source of source database the SQL statement being obtained source table by ETL Regularia.
3. visual multiple database ETL integrated approach according to claim 1, is characterized in that, is describedly optimized SQL statement and performs, obtains the ETL data of multiple source database and the object table being injected into target database comprises the following steps:
Treatment S QL statement is also optimized SQL statement according to the result of ETL rule base coupling;
The API that calling platform layer provides, the SQL statement performed after optimizing obtains ETL data and stored in data buffer, according to ETL rule base, ETL data is injected the object table of object library.
4. visual multiple database ETL integrated approach according to claim 1, is characterized in that, described treatment S QL statement being optimized SQL statement according to the result of ETL rule base coupling comprises the following steps:
SQL statement is set up a tree construction; Semantic test is carried out to each node of tree construction, and carries out cooperating measure, parsing tree is converted to the algebraic manipulation symbol tree of the inquiry plan representing initial; Algebraic manipulation symbol tree is converted to the fastest SQL sequence of execution speed.
5. visual multiple database ETL integrated system, is characterized in that comprising:
Data base management system (DBMS) layer: connect source database and target database; Mated by the ETL of source database and target database, obtain the SQL statement of the source table of source database;
Semantic layer: SQL statement is optimized and performs, obtain the ETL data of multiple source database and be injected into the object table of target database.
6. visual multiple database ETL integrated system according to claim 5, it is characterized in that, described data base management system (DBMS) layer comprises:
Graphical configuration interface: source of configuration database and target database and table name, field, and judge the type of database of source database and target database;
ETL rule base: the type of database according to source database and target database determines different ETL Regularias, then obtain SQL statement according to the source table of source database by ETL Regularia.
7. visual multiple database ETL integrated system according to claim 5, is characterized in that described semantic layer comprises:
Query compiler device: treatment S QL statement is also optimized SQL statement according to the result of ETL rule base coupling;
Enforcement engine: the API that calling platform layer provides, the SQL statement performed after optimizing obtains ETL data and stored in data buffer, according to ETL rule base, ETL data is injected the object table of object library.
8. visual multiple database ETL integrated system according to claim 1, is characterized in that described query compiler device comprises:
Query analyzer: SQL statement is set up a tree construction;
Inquiry pretreater: carry out semantic test to each node of tree construction, and carry out semantic test to each node of tree construction, and carry out cooperating measure, is converted to the algebraic manipulation symbol tree of the inquiry plan representing initial by parsing tree;
Query optimizer: algebraic manipulation symbol tree is converted to the fastest SOL sequence of execution speed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410086142.4A CN104915341B (en) | 2014-03-10 | 2014-03-10 | Visualize multiple database ETL integrated approaches and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410086142.4A CN104915341B (en) | 2014-03-10 | 2014-03-10 | Visualize multiple database ETL integrated approaches and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104915341A true CN104915341A (en) | 2015-09-16 |
| CN104915341B CN104915341B (en) | 2018-06-26 |
Family
ID=54084412
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410086142.4A Expired - Fee Related CN104915341B (en) | 2014-03-10 | 2014-03-10 | Visualize multiple database ETL integrated approaches and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104915341B (en) |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105740462A (en) * | 2016-03-02 | 2016-07-06 | 上海新炬网络信息技术有限公司 | Method for supporting data migration between different environments |
| CN106897322A (en) * | 2015-12-21 | 2017-06-27 | ***通信集团山西有限公司 | The access method and device of a kind of database and file system |
| CN107145576A (en) * | 2017-05-08 | 2017-09-08 | 科技谷（厦门）信息技术有限公司 | A kind of big data ETL for supporting visualization and procedure dispatches system |
| CN107169130A (en) * | 2017-06-08 | 2017-09-15 | 贵州优联博睿科技有限公司 | The visual inquiry method and system of a kind of database |
| CN107463709A (en) * | 2017-08-21 | 2017-12-12 | 北京奇艺世纪科技有限公司 | A kind of ETL processing method and processing devices based on multi-data source |
| CN107689982A (en) * | 2017-06-25 | 2018-02-13 | 平安科技（深圳）有限公司 | Multi-data source method of data synchronization, application server and computer-readable recording medium |
| CN107688598A (en) * | 2017-06-25 | 2018-02-13 | 平安科技（深圳）有限公司 | Source table structure analysis method, application server and computer-readable recording medium |
| CN107818368A (en) * | 2016-09-14 | 2018-03-20 | 上海翼勋互联网金融信息服务有限公司 | Risk control rule engine system on line |
| CN108062407A (en) * | 2017-12-28 | 2018-05-22 | 成都飞机工业（集团）有限责任公司 | A kind of project visualizes management and control data pick-up method |
| CN108446299A (en) * | 2018-01-25 | 2018-08-24 | 链家网（北京）科技有限公司 | The method and device of data-optimized calculating in a kind of task |
| CN109063005A (en) * | 2018-07-10 | 2018-12-21 | 阿里巴巴集团控股有限公司 | A kind of data migration method and system, storage medium, electronic equipment |
| CN109582723A (en) * | 2018-11-30 | 2019-04-05 | 深圳市思迪信息技术股份有限公司 | Distributed ETL collecting method and device |
| CN109669983A (en) * | 2018-12-27 | 2019-04-23 | 杭州火树科技有限公司 | Visualize multi-data source ETL tool |
| CN110727729A (en) * | 2018-06-29 | 2020-01-24 | 贵州白山云科技股份有限公司 | Method and device for realizing intelligent operation |
| CN110990482A (en) * | 2019-11-11 | 2020-04-10 | 中国建设银行股份有限公司 | Data synchronization method and device between asynchronous databases |
| CN111782653A (en) * | 2020-06-30 | 2020-10-16 | 平安国际智慧城市科技股份有限公司 | Data query method and device, electronic equipment and storage medium |
| CN112434059A (en) * | 2021-01-26 | 2021-03-02 | 腾讯科技（深圳）有限公司 | Data processing method, data processing device, computer equipment and storage medium |
| CN112527815A (en) * | 2020-12-02 | 2021-03-19 | 平安医疗健康管理股份有限公司 | Script migration method and device for database, computer equipment and storage medium |
| CN112783923A (en) * | 2020-11-25 | 2021-05-11 | 辽宁振兴银行股份有限公司 | Implementation method for efficiently acquiring database based on Spark and Impala |
| CN113792098A (en) * | 2021-08-02 | 2021-12-14 | 中国城市规划设计研究院 | Database SQL (structured query language) imaging-based big data visualization method, system and medium |
| CN113934786A (en) * | 2021-09-29 | 2022-01-14 | 浪潮卓数大数据产业发展有限公司 | Implementation method for constructing unified ETL |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102043841A (en) * | 2010-12-10 | 2011-05-04 | 上海市城市建设设计研究院 | Multi-source information supplying method based on Web technology and integrated service system thereof |
| US20120030172A1 (en) * | 2010-07-27 | 2012-02-02 | Oracle International Corporation | Mysql database heterogeneous log based replication |
| CN102915377A (en) * | 2012-11-14 | 2013-02-06 | 深圳市宏电技术股份有限公司 | Method and system for converting or synchronizing databases |
| CN103440273A (en) * | 2013-08-06 | 2013-12-11 | 北京航空航天大学 | Data cross-platform migration method and device |
-
2014
- 2014-03-10 CN CN201410086142.4A patent/CN104915341B/en not_active Expired - Fee Related
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120030172A1 (en) * | 2010-07-27 | 2012-02-02 | Oracle International Corporation | Mysql database heterogeneous log based replication |
| CN102043841A (en) * | 2010-12-10 | 2011-05-04 | 上海市城市建设设计研究院 | Multi-source information supplying method based on Web technology and integrated service system thereof |
| CN102915377A (en) * | 2012-11-14 | 2013-02-06 | 深圳市宏电技术股份有限公司 | Method and system for converting or synchronizing databases |
| CN103440273A (en) * | 2013-08-06 | 2013-12-11 | 北京航空航天大学 | Data cross-platform migration method and device |
Cited By (31)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106897322A (en) * | 2015-12-21 | 2017-06-27 | ***通信集团山西有限公司 | The access method and device of a kind of database and file system |
| CN106897322B (en) * | 2015-12-21 | 2019-10-29 | ***通信集团山西有限公司 | A kind of access method and device of database and file system |
| CN105740462A (en) * | 2016-03-02 | 2016-07-06 | 上海新炬网络信息技术有限公司 | Method for supporting data migration between different environments |
| CN107818368A (en) * | 2016-09-14 | 2018-03-20 | 上海翼勋互联网金融信息服务有限公司 | Risk control rule engine system on line |
| CN107145576A (en) * | 2017-05-08 | 2017-09-08 | 科技谷（厦门）信息技术有限公司 | A kind of big data ETL for supporting visualization and procedure dispatches system |
| CN107145576B (en) * | 2017-05-08 | 2020-06-23 | 科技谷（厦门）信息技术有限公司 | Big data ETL scheduling system supporting visualization and process |
| CN107169130A (en) * | 2017-06-08 | 2017-09-15 | 贵州优联博睿科技有限公司 | The visual inquiry method and system of a kind of database |
| CN107689982A (en) * | 2017-06-25 | 2018-02-13 | 平安科技（深圳）有限公司 | Multi-data source method of data synchronization, application server and computer-readable recording medium |
| CN107688598A (en) * | 2017-06-25 | 2018-02-13 | 平安科技（深圳）有限公司 | Source table structure analysis method, application server and computer-readable recording medium |
| CN107688598B (en) * | 2017-06-25 | 2021-02-09 | 平安科技（深圳）有限公司 | Source table structure analysis method, application server and computer readable storage medium |
| CN107689982B (en) * | 2017-06-25 | 2020-11-24 | 平安科技（深圳）有限公司 | Multi-data source data synchronization method, application server and computer readable storage medium |
| WO2019000628A1 (en) * | 2017-06-25 | 2019-01-03 | 平安科技（深圳）有限公司 | Source table structure parsing method and system, application server and computer-readable storage medium |
| CN107463709A (en) * | 2017-08-21 | 2017-12-12 | 北京奇艺世纪科技有限公司 | A kind of ETL processing method and processing devices based on multi-data source |
| CN108062407A (en) * | 2017-12-28 | 2018-05-22 | 成都飞机工业（集团）有限责任公司 | A kind of project visualizes management and control data pick-up method |
| CN108446299A (en) * | 2018-01-25 | 2018-08-24 | 链家网（北京）科技有限公司 | The method and device of data-optimized calculating in a kind of task |
| CN110727729A (en) * | 2018-06-29 | 2020-01-24 | 贵州白山云科技股份有限公司 | Method and device for realizing intelligent operation |
| CN109063005A (en) * | 2018-07-10 | 2018-12-21 | 阿里巴巴集团控股有限公司 | A kind of data migration method and system, storage medium, electronic equipment |
| CN109063005B (en) * | 2018-07-10 | 2021-05-25 | 创新先进技术有限公司 | Data migration method and system, storage medium and electronic device |
| CN109582723A (en) * | 2018-11-30 | 2019-04-05 | 深圳市思迪信息技术股份有限公司 | Distributed ETL collecting method and device |
| CN109582723B (en) * | 2018-11-30 | 2021-08-17 | 深圳市思迪信息技术股份有限公司 | Distributed ETL data acquisition method and device |
| CN109669983A (en) * | 2018-12-27 | 2019-04-23 | 杭州火树科技有限公司 | Visualize multi-data source ETL tool |
| CN110990482A (en) * | 2019-11-11 | 2020-04-10 | 中国建设银行股份有限公司 | Data synchronization method and device between asynchronous databases |
| CN111782653A (en) * | 2020-06-30 | 2020-10-16 | 平安国际智慧城市科技股份有限公司 | Data query method and device, electronic equipment and storage medium |
| CN112783923A (en) * | 2020-11-25 | 2021-05-11 | 辽宁振兴银行股份有限公司 | Implementation method for efficiently acquiring database based on Spark and Impala |
| CN112527815A (en) * | 2020-12-02 | 2021-03-19 | 平安医疗健康管理股份有限公司 | Script migration method and device for database, computer equipment and storage medium |
| CN112434059A (en) * | 2021-01-26 | 2021-03-02 | 腾讯科技（深圳）有限公司 | Data processing method, data processing device, computer equipment and storage medium |
| CN112434059B (en) * | 2021-01-26 | 2021-06-22 | 腾讯科技（深圳）有限公司 | Data processing method, data processing device, computer equipment and storage medium |
| CN113792098A (en) * | 2021-08-02 | 2021-12-14 | 中国城市规划设计研究院 | Database SQL (structured query language) imaging-based big data visualization method, system and medium |
| CN113792098B (en) * | 2021-08-02 | 2023-06-20 | 中国城市规划设计研究院 | Big data visualization method, system and medium based on database SQL (structured query language) imaging |
| CN113934786A (en) * | 2021-09-29 | 2022-01-14 | 浪潮卓数大数据产业发展有限公司 | Implementation method for constructing unified ETL |
| CN113934786B (en) * | 2021-09-29 | 2023-09-08 | 浪潮卓数大数据产业发展有限公司 | Implementation method for constructing unified ETL |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104915341B (en) | 2018-06-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104915341A (en) | Visual multi-database ETL integration method and system | |
| US10635675B2 (en) | Supporting pluggable databases with heterogeneous database character sets in a container database | |
| CN103455540B (en) | The system and method for generating memory model from data warehouse model | |
| US8260824B2 (en) | Object-relational based data access for nested relational and hierarchical databases | |
| US10372707B2 (en) | Query execution pipelining with pump operators | |
| US9471633B2 (en) | Eigenvalue-based data query | |
| US8943059B2 (en) | Systems and methods for merging source records in accordance with survivorship rules | |
| US10133776B2 (en) | Transforming a query by eliminating a subquery | |
| US11354284B2 (en) | System and method for migration of a legacy datastore | |
| US9471617B2 (en) | Schema evolution via transition information | |
| Lee et al. | Query performance of the IFC model server using an object-relational database approach and a traditional relational database approach | |
| CN102760143A (en) | Method and device for dynamically integrating executing structures in database system | |
| US11281668B1 (en) | Optimizing complex database queries using query fusion | |
| US11409741B2 (en) | Enabling data format specific database functionalities over existing data types by marking operand values | |
| US20220284005A1 (en) | Relational method for transforming unsorted sparse dictionary encodings into unsorted-dense or sorted -dense dictionary encodings | |
| Yuanyuan et al. | Distributed database system query optimization algorithm research | |
| US7051041B1 (en) | Simplified relational database extension to DBM hash tables and method for using same | |
| WO2018090557A1 (en) | Method and device for querying data table | |
| CN113934750A (en) | Data blood relationship analysis method based on compiling mode | |
| US10915541B2 (en) | Generic API | |
| Sreemathy et al. | Data validation in ETL using TALEND | |
| Fehily | SQL | |
| US20200311083A1 (en) | Generation of query execution plans | |
| Szumowska et al. | Extending HQL with plain recursive facilities | |
| Silva et al. | Logical big data integration and near real-time data analytics |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20180626 Termination date: 20200310 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |