CN112650797A - Isomerous DBMS data exchange method and system applying same - Google Patents

Isomerous DBMS data exchange method and system applying same Download PDF

Info

Publication number
CN112650797A
CN112650797A CN201910957281.2A CN201910957281A CN112650797A CN 112650797 A CN112650797 A CN 112650797A CN 201910957281 A CN201910957281 A CN 201910957281A CN 112650797 A CN112650797 A CN 112650797A
Authority
CN
China
Prior art keywords
data
converted
database
target
module
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201910957281.2A
Other languages
Chinese (zh)
Inventor
摆洪洲
叶会
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Geben Enterprise Service Co ltd
Original Assignee
Hangzhou Geben Enterprise Service Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Geben Enterprise Service Co ltd filed Critical Hangzhou Geben Enterprise Service Co ltd
Priority to CN201910957281.2A priority Critical patent/CN112650797A/en
Publication of CN112650797A publication Critical patent/CN112650797A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G06F16/258—Data format conversion from or to a database
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
    • G06F21/31—User authentication
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/604—Tools and structures for managing or administering access control systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
    • G06F21/6227—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database where protection concerns the structure of data, e.g. records, types, queries
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F2221/21—Indexing scheme relating to G06F21/00 and subgroups addressing additional information or applications relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F2221/2141—Access rights, e.g. capability lists, access control lists, access tables, access matrices

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Security & Cryptography (AREA)
  • Software Systems (AREA)
  • Computer Hardware Design (AREA)
  • Data Mining & Analysis (AREA)
  • Health & Medical Sciences (AREA)
  • Bioethics (AREA)
  • General Health & Medical Sciences (AREA)
  • Automation & Control Theory (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a data exchange method of isomerous DBMS and a system applying the method, wherein the method comprises the following steps: sending connection information to a source database, wherein the connection information comprises authentication information of the source database; under the condition that the authentication information is correct, extracting and exporting data to be converted in a source database according to a preset extraction rule and storing the data in a memory of a computer; converting data to be converted in a memory of a computer into target data which can be identified by a target database; and loading the converted target data to a target database. The invention can help to reduce the maintenance cost of the data import and export tool and improve the speed of data exchange among different databases; and the timeliness of conversion is improved in the conversion process, and the integrity and accuracy of target data are ensured.

Description

Isomerous DBMS data exchange method and system applying same
Technical Field
The invention relates to the technical field of computer data processing, in particular to a data exchange method of a same-structure DBMS and a system applying the method.
Background
A heterogeneous database system is a collection of related multiple database systems, including a collection of relational and non-relational databases. The heterogeneous database systems can realize the sharing and transparent access of data, and each database system already exists before being added into the heterogeneous database system and has own DBMS. Each component of the heterogeneous database has autonomy, and each database system still has application characteristics, integrity control and security control of the database while realizing data sharing.
How to realize better and faster interaction of data among heterogeneous databases and ensure timeliness becomes a technical problem to be solved by the technical personnel in the field.
Disclosure of Invention
The invention aims to provide a data exchange method of a same-structure DBMS and a system applying the method, so as to overcome the defects in the prior art.
In order to achieve the technical purpose, the technical scheme of the invention is realized as follows:
a method of data exchange for a homogeneous DBMS, the method comprising:
1) sending connection information to a source database, wherein the connection information comprises authentication information of the source database;
2) under the condition that the authentication information is correct, extracting and exporting data to be converted in a source database according to a preset extraction rule and storing the data in a memory of a computer;
3) converting the data to be converted in the memory of the computer into target data which can be identified by a target database;
4) and loading the converted target data to the target database.
Further, the authentication information includes a user name and a password.
Further, the data to be converted is cleaned according to a preset cleaning rule before the step 3).
Further, the predetermined cleaning rule is to filter spaces and/or unrecognized characters existing in each field in the data to be converted.
Further, step 3) comprises:
3.1) setting a separator for the data to be converted;
3.2) according to the separator, converting the data to be converted in a batch mode.
Further, step 3.2) comprises:
setting the number of lines of the batch processing according to the size of the data to be converted, and converting the data to be converted in each line;
and storing the converted target data.
A system for applying the method as described above, the system comprising:
the verification module is used for sending connection information to the source database, wherein the connection information comprises authentication information of the source database;
the extraction module is used for extracting and exporting the data to be converted in the source database and storing the data in the memory of the computer according to a preset extraction rule under the condition that the authentication information is correct;
the conversion module is used for converting the data to be converted in the memory of the computer into target data which can be identified by a target database;
and the loading module is used for loading the converted target data to the target database.
The data conversion device further comprises a cleaning module used for cleaning the data to be converted according to a preset cleaning rule before the data is converted by the conversion module.
Further, the conversion module includes:
the separator setting module is used for setting separators for the data to be converted;
and the batch processing module is used for converting the data to be converted in a batch processing mode according to the separators.
Further, the batch processing module comprises:
the line number setting module is used for setting the line number of the batch processing according to the size of the data to be converted and converting the data to be converted of each line;
and the storage module is used for storing the converted target data.
The invention has the beneficial effects that: the intermediate switching node is adopted to read data from one database, perform format conversion and write the data into the other database, so that data exchange can be conveniently realized among a plurality of heterogeneous databases, and the data exchange path is short; the read data is stored in a computer memory instead of a local file, so that the reading and writing speed is high; converting the exported data to be converted into target data which can be recognized by the Gemfire database in a batch processing mode, and storing the converted target data in a map set, so that the target data can be loaded into a non-relational database Gemfire in a keyvalue mode through a loading interface, and data sharing between a Greenplus database and the Gemfire database is realized; and the timeliness of conversion is improved in the conversion process, and the integrity and accuracy of target data are ensured.
Drawings
FIG. 1 is a flow chart of a method of the present invention;
FIG. 2 is a block diagram of the architecture of the system of the present invention;
FIG. 3 is a schematic diagram of a computer system running multiple databases, according to an embodiment of the present invention.
Detailed Description
The technical solution in the embodiments of the present invention is clearly and completely described below with reference to the drawings in the embodiments of the present invention.
In the embodiment of the present invention, a switching node is added among a plurality of databases, and referring to fig. 3, fig. 3 is a schematic diagram of a computer system running a plurality of databases according to an embodiment of the present invention. The databases operated by the nodes 21, 22, 23, 24, 25 are heterogeneous databases, and the intermediate node 20 is connected to the above nodes, and the above nodes may be connected or not connected. The intermediate node reads data from the database of one node and writes the data to the database of another node according to the method provided by this embodiment. The method is illustrated below with reference to fig. 3, and fig. 1 is a schematic diagram of main steps of a method for exchanging data between heterogeneous databases according to an embodiment of the present invention. In the flow shown in FIG. 1, node 20 reads data from the database of node 21 and then writes to the database of node 22. Node 21 now acts as the source database and node 22 acts as the target database.
Step 101: node 20 sends connection information to node 21. The connection information is mainly authentication information. This authentication information needs to be stored in the node 20 during configuration, and in addition, the network address of the node 21, and statements conforming to the syntax of the database in the node 21, such as statements for reading the database, etc., are stored.
Step 102: under the condition that the authentication information is correct, extracting and exporting data to be converted in the source database according to a preset extraction rule, and storing the data in a memory of the computer, wherein in the step 102, the extraction rule may be that the data to be converted is extracted and exported from the relational database according to data which needs to be extracted actually as a reference range, so as to prepare for data conversion. For the extraction of data in the relational database, different extraction modes can be adopted according to different requirements, for example: full or incremental extraction; where full extraction is analogous to data migration or data replication, which extracts the data of a table or view in a data source from a database intact. Incremental extraction refers to extracting new, modified data from tables to be extracted in the database since the last extraction.
The extraction mode may extract the data to be obtained through a keyword or a timestamp, and the extracted data is used as the data to be converted, which may be understood that the extracted data may also be extracted through an attribute value of the data in the relational database.
In the implementation of this step, the extracted data may be exported in a data-by-data mode as the data to be converted, and in this implementation, the data to be converted may be a txt file, an excel file, an xml file, or the like in a specified field format.
The relational database may also be referred to as a traditional database, which refers to a two-dimensional table model, and thus a relational database is a data organization composed of two-dimensional tables and their associations. Relational databases emphasize maintaining data integrity and consistency. Currently, the mainstream relational databases are Oracle, DB2, PostgreSQL, Microsoft SQL Server, Microsoft Access, MySQL, and wave K-DB. The relational database in the implementation mainly adopts a Greenplus database, and the Greenplus database is mainly characterized by high query speed, high data loading speed and high batch DML processing speed; and the performance can be linearly increased along with the addition of hardware, and the system has very good expandability. Therefore, it is mainly suitable for analysis-oriented applications.
Step 103: and receiving the data to be converted, and converting the data to be converted into target data which can be identified by the non-relational database.
In step 103, the data to be converted is converted in a multi-threaded batch processing mode, and the data to be converted is converted into a specified java object, so that the converted target data can be identified and processed by the non-relational database.
In this implementation, the non-relational database may be a Gemfire database, and the Gemfire database belongs to a non-traditional database. The Gemfire database is a high-performance, distributed operational data (operational data) management infrastructure located between the application cluster and the backend data source. The Gemfire provides low-delay and high-throughput data sharing and event distribution, and fully utilizes memory and disk resources in a network to form a real-time data grid. The java objects in the key-value form are stored in the Gemfire database, so that the data to be converted needs to be converted into java objects in the key-value form to form target data, thereby facilitating identification of the Gemfire database.
In order to improve timeliness of data conversion to be converted, before converting the received data to be converted, the method comprises the following steps:
setting a separator for the data to be converted;
and converting the data to be converted in a batch mode according to the separators.
When the data to be converted is exported for conversion processing, the data to be converted is required to be read line by line, the data to be converted is divided according to set separators, each line of data is processed according to the separators and is converted into a specified object, the data to be converted is converted in a batch processing mode, and the converted target data is stored in a map set.
In order to facilitate batch processing, in this embodiment, the number of lines in each batch processing may be set according to the condition of the data to be converted (e.g., size, length, etc.), and the target data objects of each batch may be stored in the map set in a specified number.
The map set provides a more general method of element storage, and is used to store pairs of elements (called "keys" and "values", i.e., key-values), where each key maps to a value.
And by adopting a multithreading batch processing mode, the timeliness of data conversion can be ensured.
In the conversion process, the setting of the separator can be only required to meet the requirement of distinguishing the separator from the data to be converted, so that the separator is prevented from colliding with the content in the data to be converted, and the same separator needs to be set for the separator of each line.
In this implementation, the separator may be set by separating with a separator between the end of each field content and the beginning of the next field content. The data to be converted extracted from the greenplus database can be subjected to corresponding field control and delimiter control.
It can be understood that, when the data to be converted further includes sub-data, the sub-data of the data to be converted is divided by using the separator; the order of the subdata can be customized according to needs, namely: and setting a separator by self-defining the sub-data, and converting the sub-data according to the arrangement sequence of the sub-data when converting the sub-data.
In this embodiment, the separator may be in a customized form.
In order to ensure that target data after conversion is more accurate, after data to be converted is derived, because characters and/or spaces and the like which cannot be recognized may exist in the data to be converted, before the data to be converted is converted, the data to be converted needs to be cleaned, and therefore, before the data to be converted is converted into the target data which can be recognized by a non-relational database, the method includes: and cleaning the data to be converted according to a preset cleaning rule.
In this implementation, the predetermined cleansing rule is to filter spaces and/or unrecognized characters existing in each field in the data to be converted. The specific cleaning process may be to delete the space and/or unrecognized character in each field of the data to be converted, or skip some abnormal information or data.
Therefore, the washing process does not substantially change the field content, so that the success rate of entering the data into the non-relational database Gemfire is improved, it should be noted that, because the washed data enters the non-relational database Gemfire, the data is in a key-value form, and the data in the form is out of order, but the integrity of the data can still be ensured.
Step 104: and loading the converted target data to the non-relational database.
In the step 104, the converted target data is stored in a map set, and the map set stores the target data in a Gemfire database by using a pull interface.
According to the data interaction method between the heterogeneous databases, provided by the invention, data in the relational database greenplus can be extracted as data to be converted, the data to be converted are exported, the exported data to be converted are subjected to batch processing conversion, and the converted target data are stored in a map set, so that the target data can be identified by the non-relational database Gemfire in a key value form, and the data sharing between the greenplus database and the Gemfire database is realized.
Corresponding to the data exchange method of the isomerous DBMS of the invention, the invention also discloses a system applying the method, as shown in figure 2, the system comprises: the system comprises:
the verification module is used for sending connection information to the source database, wherein the connection information comprises authentication information of the source database;
the extraction module is used for extracting and exporting the data to be converted in the source database and storing the data in the memory of the computer according to a preset extraction rule under the condition that the authentication information is correct;
the conversion module is used for converting the data to be converted in the memory of the computer into target data which can be identified by a target database;
and the loading module is used for loading the converted target data to the target database.
The data to be converted is cleaned according to a predetermined cleaning rule before the data is converted by the conversion module.
As a preferred embodiment, the conversion module includes:
the separator setting module is used for setting separators for the data to be converted;
and the batch processing module is used for converting the data to be converted in a batch processing mode according to the separators.
As a preferred embodiment, the batch processing module includes:
the line number setting module is used for setting the line number of the batch processing according to the size of the data to be converted and converting the data to be converted of each line;
and the storage module is used for storing the converted target data.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.

Claims (10)

1. A method for data exchange in a heterogeneous DBMS, the method comprising:
1) sending connection information to a source database, wherein the connection information comprises authentication information of the source database;
2) under the condition that the authentication information is correct, extracting and exporting data to be converted in a source database according to a preset extraction rule and storing the data in a memory of a computer;
3) converting the data to be converted in the memory of the computer into target data which can be identified by a target database;
4) and loading the converted target data to the target database.
2. The method of claim 1, wherein the authentication information comprises a username and password.
3. The method according to claim 1, wherein the data to be converted is scrubbed according to predetermined scrubbing rules before step 3).
4. The method according to claim 3, wherein the predetermined cleansing rule is to filter spaces and/or unrecognized characters existing in each field in the data to be converted.
5. The method of claim 1, wherein step 3) comprises:
3.1) setting a separator for the data to be converted;
3.2) according to the separator, converting the data to be converted in a batch mode.
6. The method according to claim 1, characterized in that step 3.2) comprises:
setting the number of lines of the batch processing according to the size of the data to be converted, and converting the data to be converted in each line;
and storing the converted target data.
7. A system for applying the method of claim 1, the system comprising:
the verification module is used for sending connection information to the source database, wherein the connection information comprises authentication information of the source database;
the extraction module is used for extracting and exporting the data to be converted in the source database and storing the data in the memory of the computer according to a preset extraction rule under the condition that the authentication information is correct;
the conversion module is used for converting the data to be converted in the memory of the computer into target data which can be identified by a target database;
and the loading module is used for loading the converted target data to the target database.
8. The system of claim 7, further comprising a cleansing module configured to cleanse the data to be converted according to a predetermined cleansing rule before the data is converted by the conversion module.
9. The system of claim 8, wherein the conversion module comprises:
the separator setting module is used for setting separators for the data to be converted;
and the batch processing module is used for converting the data to be converted in a batch processing mode according to the separators.
10. The system of claim 9, wherein the batch module comprises:
the line number setting module is used for setting the line number of the batch processing according to the size of the data to be converted and converting the data to be converted of each line;
and the storage module is used for storing the converted target data.
CN201910957281.2A 2019-10-10 2019-10-10 Isomerous DBMS data exchange method and system applying same Pending CN112650797A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910957281.2A CN112650797A (en) 2019-10-10 2019-10-10 Isomerous DBMS data exchange method and system applying same

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910957281.2A CN112650797A (en) 2019-10-10 2019-10-10 Isomerous DBMS data exchange method and system applying same

Publications (1)

Publication Number Publication Date
CN112650797A true CN112650797A (en) 2021-04-13

Family

ID=75342504

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910957281.2A Pending CN112650797A (en) 2019-10-10 2019-10-10 Isomerous DBMS data exchange method and system applying same

Country Status (1)

Country Link
CN (1) CN112650797A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113448777A (en) * 2021-07-08 2021-09-28 京东科技控股股份有限公司 Data recovery method and device

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113448777A (en) * 2021-07-08 2021-09-28 京东科技控股股份有限公司 Data recovery method and device

Similar Documents

Publication Publication Date Title
CN107506451B (en) Abnormal information monitoring method and device for data interaction
CN109522290B (en) HBase data block recovery and data record extraction method
US10331657B1 (en) Contention analysis for journal-based databases
CN106033437A (en) Method and system for processing distributed transaction
CN104133867A (en) DOT in-fragment secondary index method and DOT in-fragment secondary index system
CN111382226A (en) Database query retrieval method and device and electronic equipment
US10002142B2 (en) Method and apparatus for generating schema of non-relational database
WO2020077027A1 (en) Method and system for executing queries on indexed views
US10198346B1 (en) Test framework for applications using journal-based databases
WO2015074290A1 (en) Database implementation method
CN104239377A (en) Platform-crossing data retrieval method and device
US10133767B1 (en) Materialization strategies in journal-based databases
Kan et al. Topology modeling and analysis of a power grid network using a graph database
CN114218218A (en) Data processing method, device and equipment based on data warehouse and storage medium
CN103440265A (en) MapReduce-based CDC (Change Data Capture) method of MYSQL database
US10599614B1 (en) Intersection-based dynamic blocking
CN117112691A (en) Storage method of big data-oriented multi-storage engine database
CN103365987A (en) Clustered database system and data processing method based on shared-disk framework
CN107273462A (en) One kind builds HBase cluster full-text index methods, method for reading data and method for writing data
CN112650797A (en) Isomerous DBMS data exchange method and system applying same
CN109271456A (en) Host data library file deriving method and device
KR101955376B1 (en) Processing method for a relational query in distributed stream processing engine based on shared-nothing architecture, recording medium and device for performing the method
CN110990472B (en) Hbase-based data deriving method and Hbase-based data deriving device
CN106021574A (en) Data storage replication method and system
CN110162513A (en) Tables of data connection method and computer readable storage medium for the processing of smart grid big data

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
WD01 Invention patent application deemed withdrawn after publication

Application publication date: 20210413

WD01 Invention patent application deemed withdrawn after publication