CN111198917A

CN111198917A - Data processing method, device, equipment and storage medium

Info

Publication number: CN111198917A
Application number: CN202010009911.6A
Authority: CN
Inventors: 鲜伟
Original assignee: China Construction Bank Corp; CCB Finetech Co Ltd
Current assignee: China Construction Bank Corp
Priority date: 2020-01-06
Filing date: 2020-01-06
Publication date: 2020-05-26

Abstract

The embodiment of the invention discloses a data processing method, a data processing device, data processing equipment and a storage medium. The method comprises the following steps: acquiring a first database to which original data to be queried belong and a second database to which a mode to be queried belongs; acquiring original data of the first database; the original data is data to be stored or stored in the first database; converting the original data from a first data format supported by the first database to a second data format supported by the second database according to the data format conversion relation of different databases; and storing the conversion result into the second database for the user to inquire the original data from the second database. The method and the device aim to improve the efficiency of data query and support various use scenes of the data query.

Description

Data processing method, device, equipment and storage medium

Technical Field

The embodiment of the invention relates to the technical field of big data, in particular to a data processing method, a data processing device, data processing equipment and a storage medium.

Background

With the increase of mass data in the financial industry, the traditional relational database and data warehouse face the problems of low data processing efficiency and query efficiency. Under the circumstances, big data is produced, and as the big data is made up and developed, the distributed technology becomes mature. The method can not only effectively solve the problem of data storage capacity, but also improve the data processing efficiency through a distributed computing mode and provide better technical ecology to solve the problem of data storage and data processing which cannot be solved by the traditional technology.

However, the current processing method is usually based on an Extract-Transform-Load (ETL) tool (data warehouse technology), which requires frequent read/write operations, and the IO (Input Output) pressure is relatively high, which results in relatively low data processing efficiency.

Therefore, a data processing method is needed to improve the efficiency of data query and support a plurality of usage scenarios of data query.

Disclosure of Invention

The invention provides a data processing method, a data processing device, data processing equipment and a storage medium, which aim to improve the efficiency of data query and support various use scenes of the data query.

In a first aspect, an embodiment of the present invention provides a data processing method, including:

acquiring a first database to which original data to be queried belong and a second database to which a mode to be queried belongs;

acquiring original data of the first database; the original data is data to be stored or stored in the first database;

converting the original data from a first data format supported by the first database to a second data format supported by the second database according to the data format conversion relation of different databases;

and storing the conversion result into the second database for the user to inquire the original data from the second database.

In a second aspect, an embodiment of the present invention further provides a data processing apparatus, including:

the database acquisition module is used for acquiring a first database to which the original data to be inquired belong and a second database to which the mode to be inquired belongs;

the original data acquisition module is used for acquiring original data of the first database; the original data is data to be stored or stored in the first database;

the data format conversion module is used for converting the original data from a first data format supported by the first database to a second data format supported by the second database according to the data format conversion relation of different databases;

and the conversion result storage module is used for storing the conversion result into the second database, and is used for the user to inquire the original data from the second database.

In a third aspect, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the data processing method according to any one of the embodiments of the present invention when executing the program.

In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the data processing method according to any one of the embodiments of the present invention.

The method comprises the steps of obtaining a first database to which original data to be inquired belong and a second database to which a mode to be inquired belongs; acquiring original data of the first database; converting the original data from a first data format supported by the first database to a second data format supported by the second database according to the data format conversion relation of different databases; and storing the conversion result into the second database for the user to inquire the original data from the second database. The method and the device aim to improve the efficiency of data query and support various use scenes of the data query.

Drawings

Fig. 1 is a schematic flow chart of a data processing method according to a first embodiment of the present invention;

FIG. 2 is a block diagram of a data processing system according to a first embodiment of the present invention;

fig. 3 is a flowchart illustrating a data processing method according to a second embodiment of the present invention;

fig. 4 is a schematic structural diagram of a data processing apparatus according to a third embodiment of the present invention;

fig. 5 is a schematic structural diagram of an apparatus provided in the fourth embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.

Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the steps as a sequential process, many of the steps can be performed in parallel, concurrently or simultaneously. In addition, the order of the steps may be rearranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.

Example one

Fig. 1 is a schematic flow chart of a data processing method according to an embodiment of the present invention, which is applicable to a case where different databases are transformed, and in particular, refer to a structural framework of a data processing system shown in fig. 2, where two databases in the data processing system can be transformed. A data processing method in this embodiment may be executed by a data processing apparatus, which may be implemented in software and/or hardware, and may be integrated in an electronic device, and specifically includes the following steps:

s110, a first database to which the original data to be inquired belong and a second database to which the mode to be inquired belongs are obtained.

In this embodiment, the original data to be queried refers to data required by the user, and the data may be: the system comprises bank product transaction flow, bank product basic information, bank transaction mechanism information, bank client position information, bank product type information, bank transaction channel information, bank client flow information, bank client automobiles, bank client house property information and bank client asset falling information. The first database and the second database refer to databases storing data, and those skilled in the art should understand that the first database and the second database are interchangeable in this embodiment. Wherein the first database and the second database may be: hive, HBase, elastic search, Solr, etc.

The Hive is a set of data warehouse analysis system constructed based on Hadoop, and provides rich SQL query modes to analyze data stored in a Hadoop distributed file system. Hive can map the structured data file into a database table and provide a complete SQL query function; further, SQL statements may be converted into MapReduce task runs, and needed content may be analyzed through own SQL query, but Hive is not suitable for online (online) transaction processing, nor provides a real-time query function, which is most suitable for batch processing based on a large amount of immutable data. Therefore, when the online transaction processing and the real-time query function are needed, the data stored in the Hive needs to be converted into a database supporting the online transaction processing and the real-time query function.

The HBase is a distributed and column-oriented open source database, and all data files in the HBase are stored in a Hadoop HDFS file system. HBase supports batch operation processing and prefix query, and does not support online transaction processing.

The ElasticSearch is a distributed, high-expansion and high-real-time search and data analysis engine. It can conveniently make a large amount of data have the capability of searching, analyzing and exploring. The Elasticsearch is distributed, meaning that the index can be divided into shards, each shard can have 0 or more copies. Related data is typically stored in the same index, which consists of one or more master slices and zero or more copy slices, and once the index is created, the number of master slices cannot be changed. The ElasticSearch supports the functions of online transaction processing and real-time query, and supports the fuzzy retrieval and query, so that data can be queried in the ElasticSearch when a user needs the fuzzy retrieval and query.

Solr is a stand-alone enterprise-level search application server that provides an API (application programming Interface) to the outside. The user can submit a file with a certain format to the Solr by sending a request to generate the index. Solr supports online transaction processing and real-time query functions.

Thus, when the first database is Hive, the user queries in an ambiguous search, then the second database is elastic search, and so on.

S120, acquiring original data of the first database; the original data is data to be stored or stored in the first database.

In this embodiment, the data to be stored may be related data information imported from outside, and may be table data information. Different modes of acquiring the original data and storing the original data are different, for example, when the first database is Hive, a partial storage format is generally adopted to reduce the space utilization rate, the partial storage format can be used to enable the compression rate to reach 1/10, and the IO read-write pressure is reduced in a calculation formula.

When the first database is the HBase, a data storage file is generated in a BulkLoad and Bulkdelete mode, and then the data storage file is registered in a data management module of the HBase. The method can not cause frequent reading and writing of the memory of the HBase, and avoids the memory generated by the traditional mode operation, thereby influencing the service on the HBase.

When the first database is the ElasticSearch, the ElasticSearch is written in a bulk mode, and the mode can efficiently write the ElasticSearch.

When the first database is Solr, the data is updated in batch through the data list of Solr, and the Solr can be efficiently read and written by the method.

S130, according to the data format conversion relation of different databases, converting the original data from a first data format supported by the first database to a second data format supported by the second database.

In this embodiment, the first data format is a data format supported in the first database, and the second data format is a data format supported in the second database. When the original data needs to be queried in the second database, the original data needs to be converted from the first data format supported by the first database into the second data format supported by the second database. Specifically, data format conversion can be provided through a Spark computing engine, wherein Apache Spark is a fast and general computing engine specially designed for large-scale data processing, and now an ecosystem which develops and is widely applied at a high speed is formed. Spark is used to build large, low-latency data analysis applications, which extend the widely used MapReduce computational model. More computing modes, including interactive queries and stream processing, are efficiently supported. And the Spark uses yarn to control resources, adopts uniform hoc parameter configuration and converts the first data format into the second data format.

Optionally, converting the original data from a first data format supported by the first database to a second data format supported by the second database according to a data format conversion relationship of different databases, including:

converting the original data from the first data format supported by the first database into a universal data format according to a conversion relation between the first data format and the universal data format;

and converting the original data in the universal data format into the second data format according to the conversion relation between the second data format and the universal data format.

In this embodiment, the common data format may be converted into different data formats, where the common data format may be a DataFrame, which represents a matrix data table and has a row index and a column index. The first data format is converted into a general data format through a Spark calculation engine, and then the Spark calculation engine converts the general data format into a second data format.

S140, storing the conversion result into the second database for the user to inquire the original data from the second database.

In this embodiment, when the conversion result is stored in the second database, the user can retrieve data from the second database in the data format supported by the second database.

Example two

Fig. 3 is a schematic flow chart of a data processing method according to an embodiment of the present invention, where the embodiment is applicable to conversion of different databases, and may be executed by a data processing apparatus, where the apparatus may be implemented in a software and/or hardware manner, and may be integrated in an electronic device, and specifically includes the following steps:

s310, a first database to which the original data to be inquired belong and a second database to which the mode to be inquired belongs are obtained.

S320, acquiring original data of the first database; the original data is data to be stored or stored in the first database.

And S330, acquiring data description information in the original data.

Optionally, the obtaining of the data description information in the original data includes:

and if the original data does not carry the data description information, acquiring the data description information from a description information database.

In this embodiment, the data description information is a description of original data, and may be, for example, row and column information of data. The data description information may be carried in the original data, or the description information of the original data information may be stored in the description information database. And if the original data does not carry the data description information, acquiring the data description information from the description information database.

S340, determining whether the original data is abnormal or not according to the data description information.

And comparing the original data with the data description information, judging whether the data description information is consistent with the original data, if not, indicating that the data is abnormal data, outputting and displaying the abnormal data to a user, and reminding the user that the original data is abnormal. Specifically, whether the original data is abnormal or not can be detected through fixed length and non-fixed length, whether illegal characters exist in the original data or not is judged, and if the illegal characters exist, the illegal characters are deleted.

S350, converting the original data from a first data format supported by the first database to a second data format supported by the second database according to the data format conversion relation of different databases.

And S360, storing the conversion result into the second database for the user to inquire the original data from the second database.

According to the embodiment of the invention, the original data is compared with the data description information of the original data to judge whether abnormal original data exists, and if the abnormal original data exists, the abnormal data is output, so that the accuracy of data query can be improved.

EXAMPLE III

Fig. 4 is a schematic structural diagram of a data processing apparatus according to a third embodiment of the present invention. The data processing device provided by the embodiment of the invention can execute the data processing method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method. As shown in fig. 4, the apparatus includes:

a database obtaining module 401, configured to obtain a first database to which original data to be queried belongs and a second database to which a manner to be queried belongs;

a raw data obtaining module 402, configured to obtain raw data of the first database; the original data is data to be stored or stored in the first database;

a data format conversion module 403, configured to convert, according to a data format conversion relationship of different databases, the original data from a first data format supported by the first database into a second data format supported by the second database;

a conversion result storage module 404, configured to store the conversion result in the second database, so that the user can query the original data from the second database.

The data format conversion module 403 is specifically configured to:

The device, still include:

a data description information obtaining module 405, configured to obtain data description information in the original data;

a data anomaly determination module 406, configured to determine whether the original data is anomalous according to the data description information.

The data description information obtaining module 405 is specifically configured to obtain the data description information from a description information database if the original data does not carry the data description information.

It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the above-described apparatus may refer to the corresponding process in the foregoing method embodiment, and is not described herein again.

Example four

Fig. 5 is a schematic structural diagram of an apparatus according to a fourth embodiment of the present invention, and fig. 5 is a schematic structural diagram of an exemplary apparatus suitable for implementing the embodiment of the present invention. The device 12 shown in fig. 5 is only an example and should not bring any limitations to the functionality and scope of use of the embodiments of the present invention.

As shown in FIG. 5, device 12 is in the form of a general purpose computing device. The components of device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.

Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, micro-channel architecture (MAC) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

Device 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by device 12 and includes both volatile and nonvolatile media, removable and non-removable media.

The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. Device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 5, and commonly referred to as a "hard drive"). Although not shown in FIG. 5, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.

A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of embodiments described herein.

Device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), with one or more devices that enable a user to interact with device 12, and/or with any devices (e.g., network card, modem, etc.) that enable device 12 to communicate with one or more other computing devices. Such communication may be through an input/output (I/O) interface 22. Also, the device 12 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet) via the network adapter 20. As shown in FIG. 5, the network adapter 20 communicates with the other modules of the device 12 via the bus 18. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

The processing unit 16 executes various functional applications and data processing by executing programs stored in the system memory 28, for example, to implement a data processing method provided by an embodiment of the present invention, including:

EXAMPLE five

An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program (or referred to as a computer-executable instruction) is stored, where the computer program, when executed by a processor, can implement a data processing method according to any of the embodiments described above, and the method includes:

Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for embodiments of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data processing method, comprising:

2. The method of claim 1, wherein converting the original data from a first data format supported by the first database to a second data format supported by the second database according to a data format conversion relationship of different databases comprises:

3. The method of claim 1, wherein before converting the original data from the first data format supported by the first database to the second data format supported by the second database according to the data format conversion relationship of different databases, further comprising:

acquiring data description information in the original data;

and determining whether the original data is abnormal or not according to the data description information.

4. The method of claim 3, wherein obtaining data description information in the raw data comprises:

5. A data processing apparatus, comprising:

6. The apparatus of claim 5, wherein the data format conversion module is specifically configured to:

7. The apparatus of claim 5, further comprising:

the data description information acquisition module is used for acquiring data description information in the original data;

and the data abnormity determining module is used for determining whether the original data is abnormal or not according to the data description information.

8. The apparatus according to claim 7, wherein the data description information obtaining module is specifically configured to obtain the data description information from a description information database if the original data does not carry the data description information.

9. Computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the data processing method according to any of claims 1-4 when executing the program.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the data processing method of any one of claims 1 to 4.