CN106095991B - A method of automatically generating the code from relevant database to MongoDB database data migration - Google Patents
A method of automatically generating the code from relevant database to MongoDB database data migration Download PDFInfo
- Publication number
- CN106095991B CN106095991B CN201610454184.8A CN201610454184A CN106095991B CN 106095991 B CN106095991 B CN 106095991B CN 201610454184 A CN201610454184 A CN 201610454184A CN 106095991 B CN106095991 B CN 106095991B
- Authority
- CN
- China
- Prior art keywords
- name
- type
- code
- column
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Stored Programmes (AREA)
Abstract
The invention discloses a kind of method automatically generated from relevant database to the code of MongoDB database data migration, the following steps are included: it is Step 1: read operating parameter from configuration file, and metadata is read from relevant database；Step 2: generating model according to metadata；Step 3: customizing model according to demand；Step 4: according to model and template generation code.The purpose of the present invention is auxiliary program persons to carry out from relevant database to MongoDB database migration, the mainly migration in data.Compared to script is write certainly, the present invention converts reprinting (ETL) tool, the auxiliary that the present invention also has partial code to migrate except Data Migration with more generality, compared to extracting.
Description
Technical field
The present invention relates to a kind of sides automatically generated from relevant database to the code of MongoDB database data migration
Method belongs to the fields such as database migration and software auxiliary development.
Background technique
Claimed according to a report of EMC Inc., total amount of data in the world is just increased with annual 40% speed, to 2020
Year, it will reach 44ZB.In this context, big data has attracted more and more concerns.Big data refers to that a kind of scale is big
To the data acquisition system for being well beyond traditional database software means capability range in terms of acquisition, storage, management, analysis.
Relevant database is built upon the database on the basis of relational model, by means of mathematical concepts such as algebras of sets and
Method handles the data in database.Although traditional relevant database is stablized by the mature of many years, face
When emerging big data, its advantages, such as transaction consistency and powerful sql query capability, it is difficult to it embodies.And relationship type number
According to the deficiency in library, scalability and availability are low, high concurrent read-write performance is low is but amplified.
In order to solve the bottleneck that relevant database encounters, people begin one's study the advanced database skill under big data background
Art, NoSQL are exactly one such.NoSQL storage organization is simple, is comparing traditional database in big data mass memory
There is the advantage in performance.And Document image analysis MongoDB is exactly the outstanding person in NoSQL.With high-performance, easily deployment,
It easily uses, the very convenient feature of storing data.
Summary of the invention
In many situations, using can be by being improved efficiency to the migration of NoSQL database.
The purpose of the present invention is auxiliary program persons to carry out from relevant database to MongoDB database migration, mainly
Migration in data.Compared to script is write certainly, the present invention converts (ETL) tool of reprinting with more generality, compared to extracting, this
The auxiliary that invention also has partial code to migrate except Data Migration.
The technical solution of the present invention is as follows: one kind is automatically generated from relevant database to MongoDB database data migration
Code method, the code of generation comprises the steps of:
1) operating parameter is read from configuration file, and metadata is read from relevant database；
2) model is generated according to metadata；
3) customizing model according to demand；
4) according to model and template generation code.
The specific steps of step 1 include:
1.1) parameter is read from configuration file, configuration file is defined by the user, and the content for needing to define includes: to generate generation
The relevant parameter of code, relevant database and MongoDB；
1.2) table name that all tables (table) is read from relevant database reads first number of every table according to table name
According to.The metadata of reading includes each in index (index) information of table, external key (foreign key) information of table and table
Arrange the column name of (column).The metadata of column is read according to column name, includes column name, the type of column, decimal point precision.
The specific steps of step 2 include:
2.1) model is generated according to metadata, model metadata as acquired in previous step generates, and the type of model is table
(list)。
2.2) conversion is named to the table name in Table class, the column in every table (table) is traversed, in Column class
The name conversion of column name is obtained the table name used in code and column name, and the method for naming conversion is as follows: original name is converted to
Underscore is removed initial caps by small letter, and first letter after underscore is capitalized；
2.3) column in every table (table) are traversed, the data type in Column class are converted, transform method is such as
Under: if usage type short when type is TINYINT, SMALLINT；Usage type int when if type is INTEGER；
Usage type BigInteger when if type is BIGINT；Usage type float when if type is REAL, FLOAT；If
Usage type double when type is DOUBLE；If type is DECIMAL, NUMERIC, if decimal point precision is 0, use
Type B igInteger, otherwise usage type BigDecimal；Usage type boolean when if type is BIT, BOOLEAN；
Usage type String when if type is CHAR, VARCHAR, LONGVARCHAR；If type be BINARY,
Usage type byte [] when VARBINARY, LONGVARBINARY；Usage type Date when if type is DATE；Such as fruit
Usage type Time when type is TIME；Usage type Timestamp when if type is TIMESTAMP；Otherwise usage type
Object。
The specific steps of step 3 include:
3.1) whether confirmation table name needs in addition to give a name, if it is desired, points out and modify table name, table name cannot repeat；
3.2) mode of confirmation processing connection table is embedded or quotes, and default behavior is reference, if using embedded
Mode handles connection table, only one layer nested；
3.3) it is confirmed whether have additional table to need to connect.It is the table for having foreign key relationship that default, which needs the table connected,.If
It needs, it is indicated that the table name and column name of connection table and connected table；
3.4) confirm whether the column name of every table needs in addition to give a name, if it is desired, point out table name and column name, and modify column
Name, column name cannot repeat；
3.5) confirm every table method whether in need additionally generated.It is defaulted as each individual character section (column) positive sequence and generates one
A method.If necessary, it is indicated that method name, method name cannot repeat, which field name, sortord.If necessary to more complicated
Method, provide implementation.
The specific steps of step 4 include:
It 4.1) is the generation of configuration file first according to model and template generation code, configuration file including engineering is deposited
Put the file of MongoDB configuration；
4.2) every table (table) in model is traversed, the corresponding entity (entity) of table is generated and obtains data side
The library (Repository) of method；
4.3) generate operating file: every table (table) in traversal model generates the code of insertion table, generates insertion
The code of index generates the code quoted in insertion table.
The invention has the following beneficial effects:
1) convenient: to automatically generate for migration use, to a certain extent customized code, mitigate manual operation.It generates
The corresponding entity of table and method base, facilitate subsequent code migration；
2) MongoDB specialization: compared to General ETL Tool, process provides the supports to some additional features, such as
The adoptable way of reference connection table of MongoDB, supports the migration to MongoDB so much the betterly.
Detailed description of the invention
Fig. 1 is the flow chart that data type shown in step 2.3 carries out conversion process.
Fig. 2 is the broad flow diagram that the present invention is implemented.
Specific embodiment
Below in conjunction with specific embodiment, the invention will be further described.But the present invention is not limited to following implementation.
This example generates a maven engineering based on Spring Data frame.Fig. 2 is the broad flow diagram that the present invention is implemented.
First according to configuration file connection relationship type database, the driving of connection is depending on type of database.If it is
Oracle database then uses ojdbc, then uses mysql-connector-java, etc. if it is MySql database.Then
The table name of all tables (table) is read from relevant database.As read shown in metadata 202 in Fig. 2, read according to table name
The metadata of every table, the column name of each column in the foreign key information and table of index information, table including table.It is read according to column name
The metadata of column.The method of reading uses the jdbc method for driving institute's band.
Model is generated then according to metadata, model is the table (list) that element type is Table class.As in Fig. 2 to table
Name carries out shown in conversion 204, names conversion to the table name in Table class, the column in every table (table) is traversed, to Column
Column name name conversion in class is obtained the table name used in code and column name.Name after conversion is in Java code
It uses.Needing the reason of converting is to need since the requirement in different persistence frameworks to name is different using after a set of conversion
Name adapt to more kinds of frames.The code based on Spring Data frame is generated in this example, in Spring
In Data underscore have special meaning, so in column name contain underscore when, then using find+ column name as method name when
Original meaning can be distorted.As carried out the column in every table being traversed, in Column class shown in conversion 205 in Fig. 2 to the data type of column
Data type converted.The reason of carrying out data type conversion is that the data type in sql and java is not quite similar.
Fig. 1 is the flow chart that data type shown in step 2.3 carries out conversion process.
As shown in the mode 207 of confirmation processing connection table in Fig. 2, the mode of confirmation processing connection table is still drawn to be embedded
With this example is reference.Since NoSql database does not support attended operation generally, connection is used in former relevant database
It needs manually to connect table when place moves to NoSql database.MongoDB recommends by the way of embedded or reference
Connection is specifically depended on the circumstances using which kind of mode.It is embedded to have better inquiry velocity compared to reference, however have hard disk additional
Expense, and there are more copies.Reference possesses higher flexibility than embedded, however can make the efficiency drop of inquiry
It is low.
According to model and template generation code, template is the code snippet of predefined keyword vacancy, according to demand not
With predefined different template, generating process is first to select template according to demand；According to model completion template；By the mould after completion
Plate combination producing code.It is used in this example using way of reference processing connection table and using the mould of Spring Data frame
Plate.Built-up pattern and template use tool freemarker, freemarker be it is a kind of based on template and the data to be changed come
Generate the general utility tool of output text.
Code after generation includes three parts, first is that configuration file, configuration file pom.xml including maven engineering, is deposited
Put the file MongoConfig.java of MongoDB configuration.Second is that the entity packet of data and relevant to Spring Data
Repository packet.Third is that operating file Main.java.
Main.java is run, data are moved into MongoDB database by relevant database.The entity packet of data with
And Repository packet can be used for code migration.
Above embodiment is merely illustrative of the invention's technical idea, and this does not limit the scope of protection of the present invention, all
It is any changes made on the basis of the technical scheme according to the technical idea provided by the invention, each falls within present invention protection model
Within enclosing.The technology that the present invention is not directed to can be realized by existing technology.
Claims (2)
1. a kind of method automatically generated from relevant database to the code of MongoDB database data migration, feature exist
In, comprising the following steps:
Step 1: reading operating parameter from configuration file, and metadata is read from relevant database；
The specific steps of the step 1 include:
Step 1.1 reads parameter from configuration file, and configuration file is defined by the user, and the content for needing to define includes: to generate generation
The relevant parameter of code, relevant database and MongoDB；
Step 1.2 reads the table name of all tables (table) from relevant database, and first number of every table is read according to table name
According to；The metadata of reading includes each in index (index) information of table, external key (foreign key) information of table and table
Arrange the column name of (column)；The metadata of column is read according to column name, includes column name, the type of column, decimal point precision；
Step 2: generating model according to metadata；
Include: according to the specific steps that metadata generates model in the step 2
Step 2.1 generates model according to metadata, and model metadata as acquired in previous step generates, and the type of model is table
(list)；
Step 2.2 names conversion to the table name in Table class, the column in every table (table) is traversed, in Column class
The name conversion of column name is obtained the table name used in code and column name, and the method for naming conversion is as follows: original name is converted to
Underscore is removed initial caps by small letter, and first letter after underscore is capitalized；
Step 2.3 traverses the column in every table (table), converts to the data type in Column class, and transform method is such as
Under: if usage type short when type is TINYINT, SMALLINT；Usage type int when if type is INTEGER；
Usage type BigInteger when if type is BIGINT；Usage type float when if type is REAL, FLOAT；If
Usage type double when type is DOUBLE；If type is DECIMAL, NUMERIC, if decimal point precision is 0, use
Type B igInteger, otherwise usage type BigDecimal；Usage type boolean when if type is BIT, BOOLEAN；
Usage type String when if type is CHAR, VARCHAR, LONGVARCHAR；If type be BINARY,
Usage type byte [] when VARBINARY, LONGVARBINARY；Usage type Date when if type is DATE；Such as fruit
Usage type Time when type is TIME；Usage type Timestamp when if type is TIMESTAMP；Otherwise usage type
Object；
Step 3: customizing model according to demand；
The specific steps of customizing model include: according to demand in the step 3
Whether step 3.1 confirmation table name needs in addition to give a name, if it is desired, points out and modify table name, table name cannot repeat；
The mode of step 3.2 confirmation processing connection table is embedded or quotes that default behavior is reference, if using embedded
Mode handles connection table, only one layer nested；
Step 3.3 is confirmed whether have additional table to need to connect, and it is the table for having foreign key relationship that default, which needs the table connected,；If
It needs, it is indicated that the table name and column name of connection table and connected table；
Step 3.4 confirms whether the column name of every table needs in addition to give a name, if it is desired, points out table name and column name, and modify column
Name, column name cannot repeat；
Step 3.5 confirms every table method whether in need additionally generated, is defaulted as each individual character section (column) positive sequence and generates one
A method；If necessary, it is indicated that method name, method name cannot repeat, which field name, sortord；If necessary to more complicated
Method, provide implementation；
Step 4: according to model and template generation code；
Include: according to the specific steps of model and template generation code in the step 4
Step 4.1 is the generation of configuration file first according to model and template generation code, and configuration file including engineering is deposited
Put the file of MongoDB configuration；
Step 4.2 traverses every table (table) in model, generates the corresponding entity (entity) of table and obtains data side
The library (Repository) of method；
Step 4.3 generates operating file: every table (table) in traversal model generates the code of insertion table, generates insertion
The code of index generates the code quoted in insertion table；
According to model and template generation code, template is the code snippet of predefined keyword vacancy, different pre- according to demand
Different templates is defined, generating process is first to select template according to demand；According to model completion template；By the template group after completion
Symphysis is at code；Using using way of reference processing connection table and use the template of Spring Data frame；Built-up pattern
Using tool freemarker, the freemarker with template is that a kind of generated based on template and the data to be changed is exported
The general utility tool of text.
2. the code according to claim 1 automatically generated from relevant database to MongoDB database data migration
Method, which is characterized in that template described in step 4 be predefined keyword vacancy code snippet.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610454184.8A CN106095991B (en) | 2016-06-21 | 2016-06-21 | A method of automatically generating the code from relevant database to MongoDB database data migration |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610454184.8A CN106095991B (en) | 2016-06-21 | 2016-06-21 | A method of automatically generating the code from relevant database to MongoDB database data migration |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106095991A CN106095991A (en) | 2016-11-09 |
| CN106095991B true CN106095991B (en) | 2019-07-09 |
Family
ID=57237859
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610454184.8A Active CN106095991B (en) | 2016-06-21 | 2016-06-21 | A method of automatically generating the code from relevant database to MongoDB database data migration |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106095991B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW201820175A (en) * | 2016-11-22 | 2018-06-01 | 財團法人資訊工業策進會 | Data base transformation server and data base transformation method thereof |
| US10095511B1 (en) | 2017-02-23 | 2018-10-09 | Amdocs Development Limited | System, method, and computer program for converting a current Java project to a Maven project |
| CN108038182A (en) * | 2017-12-08 | 2018-05-15 | 浪潮软件股份有限公司 | A kind of method and apparatus of the relation data library storage data based on Flume |
| CN110716913B (en) * | 2019-07-31 | 2022-12-20 | 安徽四创电子股份有限公司 | Mutual migration method of Kafka and Elasticissearch database data |
| CN110851425B (en) * | 2019-11-18 | 2023-05-26 | 上海新炬网络技术有限公司 | Method for migrating Oracle data to Mongo DB database based on Mongify |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103530327A (en) * | 2013-09-25 | 2014-01-22 | 清华大学深圳研究生院 | Method for migrating data from non-relational database to relational database |
| CN103810275A (en) * | 2014-02-13 | 2014-05-21 | 清华大学 | Method and device for data interaction between non-relation type database and relation type database |
| CN104123392A (en) * | 2014-08-11 | 2014-10-29 | 吉林禹硕动漫游戏科技股份有限公司 | Tool and method for transferring relational database to HBase |
| CN105279285A (en) * | 2015-11-23 | 2016-01-27 | 上海斐讯数据通信技术有限公司 | Synchronization system and method of relation-type database and non-relation-type database |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160055233A1 (en) * | 2014-08-25 | 2016-02-25 | Ca, Inc. | Pre-join tags for entity-relationship modeling of databases |
-
2016
- 2016-06-21 CN CN201610454184.8A patent/CN106095991B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103530327A (en) * | 2013-09-25 | 2014-01-22 | 清华大学深圳研究生院 | Method for migrating data from non-relational database to relational database |
| CN103810275A (en) * | 2014-02-13 | 2014-05-21 | 清华大学 | Method and device for data interaction between non-relation type database and relation type database |
| CN104123392A (en) * | 2014-08-11 | 2014-10-29 | 吉林禹硕动漫游戏科技股份有限公司 | Tool and method for transferring relational database to HBase |
| CN105279285A (en) * | 2015-11-23 | 2016-01-27 | 上海斐讯数据通信技术有限公司 | Synchronization system and method of relation-type database and non-relation-type database |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106095991A (en) | 2016-11-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106095991B (en) | A method of automatically generating the code from relevant database to MongoDB database data migration | |
| CN110929042B (en) | Knowledge graph construction and query method based on power enterprise | |
| Li | Transforming relational database into HBase: A case study | |
| CN108805510A (en) | A kind of construction drawing design BIM model compliance checking methods and system | |
| CN106126486B (en) | Temporal information coding method, encoded radio search method, coding/decoding method and device | |
| CN103853820A (en) | Data processing method and data processing system | |
| CN102566990B (en) | Method and device for performing data manipulation in Java application | |
| CN109472446B (en) | BIM model-based engineering budget estimate planning method | |
| CN104765872A (en) | Fast image retrieval method based on integrated hash encoding | |
| CN102999524B (en) | A kind of document associations search method and system | |
| WO2015074290A1 (en) | Database implementation method | |
| CN107491476B (en) | Data model conversion and query analysis method suitable for various big data management systems | |
| CN104504001A (en) | Massive distributed relational database-oriented cursor creation method | |
| CN101093513A (en) | Method and system for solving issue of supporting multilanguage of computer software | |
| CN104021156A (en) | Method for rapidly collecting multi-layer fact data based on SQL statements | |
| CN103246753A (en) | Method for generating entity metadata model according to database structure | |
| CN110737729A (en) | Engineering map data information management method based on knowledge map concept and technology | |
| CN101751400A (en) | System and method of technical data analysis and system of patent analysis | |
| CN110874717B (en) | Data management method | |
| CN103198141A (en) | Data record access control method and device in hierarchical relationship | |
| Pan et al. | A Novel HDF‐Based Data Compression and Integration Approach to Support BIM‐GIS Practical Applications | |
| WO2021233478A1 (en) | Image operation information hierarchical coding and tracing method | |
| CN105550176A (en) | Basic mapping method for relational database and XML | |
| CN113254517A (en) | Service providing method based on internet big data | |
| CN110716913B (en) | Mutual migration method of Kafka and Elasticissearch database data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20200624 Address after: No. 41-1, Qianfo Shandong Road, Lixia District, Jinan City, Shandong Province Patentee after: SHANDONG CIVIC SE COMMERCIAL MIDDLEWARE Co.,Ltd. Address before: No. 163 Qixia Xianlin Avenue District of Nanjing City, Jiangsu province 210046 Patentee before: NANJING University |