CN111382207B

CN111382207B - Data processing method, device, system and storage medium

Info

Publication number: CN111382207B
Application number: CN202010208117.4A
Authority: CN
Inventors: 赵立才; 陈军; 武想益; 杨晗琦
Original assignee: China Construction Bank Corp
Current assignee: China Construction Bank Corp
Priority date: 2020-03-23
Filing date: 2020-03-23
Publication date: 2023-06-27
Anticipated expiration: 2040-03-23
Also published as: CN111382207A

Abstract

The invention discloses a data processing method, a device, a system and a storage medium. The method comprises the following steps: determining the type of the received data to be processed; determining a corresponding data processing strategy according to the type of the data to be processed; and processing the data to be processed according to the data processing strategy. The embodiment of the invention processes the data to be processed by adopting a distributed and centralized fusion framework, reduces the dependence on an IBM mainframe, reduces the cost overhead of hardware resources and improves the computing capacity of the system.

Description

Data processing method, device, system and storage medium

Technical Field

Embodiments of the present invention relate to computer technology, and in particular, to a data processing method, apparatus, system, and storage medium.

Background

With the increasing popularity of internet finances, the sustained fire of electronic commerce has greatly facilitated activity, and the transaction volume and system Throughput (TPS) of debit card fast consumption has increased at 20% per year. Double eleven reached 12000TPS and 18000TPS in 2017 and 2018, respectively. 21000TPS was expected to be reached in 2019.

At present, data processing is generally deployed on a centralized platform of an IBM mainframe, and although the system has the advantages of high stability and strong overall processing capacity, the "centralized" characteristic of the system leads to limited lateral expansion capacity of the system, and the system can only be upgraded by purchasing computing resources such as a central processing unit (Central Processing Unit, CPU) and the like. However, the computing resources of the international business machines corporation (International Business Machines Corporation, IBM) mainframe are very expensive, and it is calculated that if the processing power of the system is increased by purchasing computing resources alone, several hundred million funds are spent each year against the increase of the annual transaction amount and TPS, which is a significant cost.

Disclosure of Invention

In view of this, the present invention provides a data processing method, apparatus, system, and storage medium, which reduce the cost overhead of hardware resources and improve the computing power of the system.

In one embodiment, an embodiment of the present invention provides a data processing method, including:

determining the type of the received data to be processed;

determining a corresponding data processing strategy according to the type of the data to be processed;

and processing the data to be processed according to the data processing strategy.

In one embodiment, an embodiment of the present invention further provides a data processing apparatus, including:

the first determining module is used for determining the type of the received data to be processed;

the second determining module is used for determining a corresponding data processing strategy according to the type of the data to be processed;

and the processing module is used for processing the data to be processed according to the data processing strategy.

In one embodiment, an embodiment of the present invention further provides a data processing system, including: a main system and a sub-system; the main system adopts a first type system and a first type database; the subsystem adopts a second class system and a second class database;

the host system includes: a memory, a main processor, and one or more coprocessors;

the subsystem includes: a memory, and one or more processors;

the memory is used for storing one or more programs;

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the data processing method as described in any of the embodiments above.

In an embodiment, a computer-readable storage medium has stored thereon a computer program which, when executed by a processor, implements the data processing method according to the first aspect.

The invention determines the type of the received data to be processed; determining a corresponding data processing strategy according to the type of the data to be processed; and processing the data to be processed according to the data processing strategy. The embodiment of the invention processes the data to be processed by adopting a distributed and centralized fusion framework, thereby reducing the dependence on an IBM mainframe, reducing the cost overhead of hardware resources and improving the lateral expansion capacity and the computing capacity of the system.

Drawings

FIG. 1 is a flow chart of a data processing method according to an embodiment of the present invention;

FIG. 2 is a block diagram of a data processing system according to an embodiment of the present invention;

fig. 3 is a block diagram of a processing structure of a sms EDA event according to an embodiment of the present invention;

FIG. 4 is a block diagram of a processing architecture for accounting for a running water event provided by an embodiment of the present invention;

FIG. 5 is a block diagram of a data processing apparatus according to an embodiment of the present invention;

FIG. 6 is a schematic diagram of a hardware architecture of a data processing system according to an embodiment of the present invention.

Detailed Description

The invention is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present invention are shown in the drawings.

The distributed relational database architecture (Distributed Relational Database Architecture, DRDA) technology is an IBM standard that accesses database information that conforms to the structured query language (Structured Query Language, SQL) standard across IBM platforms. DRDA technology is an important component in the IBM information warehouse framework, which defines a large backend server that clients can access through smaller workgroup-based mediating servers. Wherein, DRDA technique has the following functions: defining an interface protocol between the client and the background database; an interconnection framework for IBM's DB2, DBM, SQL/DS, and SQL/400 database systems is provided; supporting a multi-vendor provided database system; transaction (unit of work) processing on a distributed database is supported.

In DRDA, the client is called an Application Requester (ARS), the background server is called an Application Server (AS), the protocol is called an Application Support Protocol (ASP), and an interface between the AR and the AS is provided. The entire process operates on the SNA network, but is also intended to support OSI and TCP/IP. There is an additional protocol called the Database Support Protocol (DSP) that enables one AS to act AS an AR for another server. In this way the servers can talk to each other and pass requests from the client AR. The original protocol supported only one statement in the Structured Query Language (SQL) for one database, but future versions would provide multiple statements for one or more databases.

DRDA is one of the bases for establishing client/server computing in IBM environments. Other bases are advanced peer-to-peer networking (APPN) and Distributed Data Management (DDM). Through the information repository and DRDA, the IBM computer uses its mainframe computer, which is an integral part of the enterprise center, as a storage platform for various types of information, including multimedia information.

Among existing schemes, similar schemes of DRDA techniques include: read-write separation techniques and DRDA read-only access. First, read-write separation technology, through duplicating a set of database as read-only database, send read-only request to read-only database, thus has reduced the processing pressure of writing database, raise the overall performance of the system. The read-write database with separate reading and writing is adopted to simplify the complexity of system reconstruction by adopting an isomorphic database. But the core system of I's line is deployed on IBM mainframe, the environment of IBM is closed, and the use of isomorphic database still uses mainframe resources, so that the optimization purpose can not be achieved. Secondly, DRDA read-only access is performed, some inquiry transactions are deployed on a Linux platform, and a host database is accessed through a DRDA technology. This solution can only address read-only transactions and cannot address financial transactions such as quick consumption.

Meanwhile, the main system belongs to a centralized system and cannot be laterally expanded; the main system technology is closed, and advanced technologies of other platforms cannot be used; the main system has expensive resources and is not beneficial to the expansion of the system in future.

Based on the above, the embodiment of the invention provides a 'centralized + distributed' fusion framework, which reduces the dependence on an IBM mainframe, reduces the cost overhead of hardware resources, improves the computing power of the system and deals with the increase of annual transaction amount.

Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention, where the method may be implemented by a data processing apparatus, and the method may be implemented by hardware and/or software, and may be generally integrated in a data processing system, where the data processing method is applicable to a case of processing data using a centralized and distributed fusion framework. Wherein the data processing system comprises: the system comprises a main system and a sub-system, wherein the main system is a centralized system, and the sub-system is a distributed system.

As shown in fig. 1, the method specifically includes the following steps:

s110, determining the type of the received data to be processed.

In an embodiment, the data to be processed refers to data information generated in the process of performing a fast consumption transaction. Such as data information generated during debit card consumption, debit card quick payment, account balance processing, live detail processing, consumption line registration, and the like.

In one embodiment, the type of data to be processed includes one of the following: parameter class data, query class data, update class data, and newly add class data. In an embodiment, the parameter class data refers to parameters that need to be read in a fast consumption transaction, for example, the parameter class data includes: institution parameters, province parameters, product parameter tables, pricing parameter tables. The parameter class data may have the following characteristics: read-only, less variation, low timeliness requirements, etc. Query class data refers to debit cards and their account data, for example, the query class data includes: whether loss is reported, whether the account is frozen, balance inquiry and the like. Update class data refers to data generated during the updating of the balance of the debit card account. The new class data refers to data generated during the process of registering details of the debit card account and checking up the line of flow.

S120, determining a corresponding data processing strategy according to the type of the data to be processed.

In an embodiment, the data to be processed are of different data types, and the corresponding data processing strategies are also different. It is understood that different data processing strategies are adopted for processing according to the characteristics of the data to be processed of different data types.

S130, processing the data to be processed according to the data processing strategy.

In an embodiment, after determining a data processing policy for the data to be processed, the data to be processed is processed according to the data processing policy.

According to the technical scheme, the data to be processed is processed by adopting the distributed and centralized fusion framework, so that the dependence on an IBM mainframe is reduced, the cost overhead of hardware resources is reduced, and the transverse expansion capacity and the computing capacity of the system are improved.

In an embodiment, in the case that the type of the data to be processed is parameter class data, the data processing policy is a data primary and secondary synchronization mechanism of the cross-platform heterogeneous database;

correspondingly, the data to be processed is processed according to the data processing strategy, which comprises the following steps: the auxiliary system receives intermediate data synchronously updated by the main system, wherein the intermediate data is data in a first target file format obtained by archiving data to be processed by the main system; and the subsystem analyzes the intermediate data into data in a corresponding table structure file format and accesses the data in the table structure file format.

In an embodiment, the primary system employs a first type system and a first type database, and the secondary system employs a second type system and a second type database. Illustratively, the first type of system is an IBM system and the first type of database is a DB2 database; the second class of system is Linux system, and the second class of database is Oracle database.

In the embodiment, because the data size of the parameter class data is small, a data copy can be directly made on the parameter class data in the first class database of the first class system, and the data copy is synchronously updated to the second class database of the second class system, so that when the parameter class data is accessed, the second class system is directly accessed without accessing the first class system, the access rate of the first class system is reduced, and the transaction bearing capacity of the data system is further improved. Of course, in the actual operation process, whether to directly access the second type of system can be determined according to the complexity of the payment scene, for example, when the payment scene is a complex scene, the first type of system can be directly accessed; and when the payment scene is a simple scene, the second type system can be directly accessed. Illustratively, a simple scenario, which may be common debit card consumption; the complex scene may be: the debit card has overdrawn lines, and the debit card signs the scene of the subsidy. Of course, this is not limited to this, and may be adjusted according to actual conditions.

In an embodiment, in the case that the type of the data to be processed is query class data, the data processing policy is DRDA technology access;

correspondingly, the data to be processed is processed according to the data processing strategy, which comprises the following steps: receiving a data access request of a subsystem; determining a corresponding access interface according to the data access request; and determining whether to process the data to be processed by adopting a coprocessor of the main system according to the access interface.

Because the data volume of the query data is relatively large and the update is frequent, the query data cannot be solved by adopting a data copy scheme. In an embodiment, the DB2 database of the primary system is directly accessed from the secondary system using DRDA techniques. In the scenario of accessing the host system using the DRDA technique, the host system uses a coprocessor to process the query class data to reduce the pressure of the host processor. In order to reduce the hardware cost of the data processing system during actual operation, the main system may include: a plurality of host processors, and a plurality of coprocessors.

In one embodiment, where the type of data to be processed is update class data, the data processing policy is an Event-driven architecture (EDA) mechanism;

correspondingly, the data to be processed is processed according to the data processing strategy, which comprises the following steps: the auxiliary system acquires a target element of update type data in the main system; synchronizing and storing the target elements to a target database of the subsystem itself; and under the condition that the EDA sending thread is detected, sending the target elements in the target database to a target message center so that the subscriber can further process the data.

In an embodiment, in the case where the data to be processed is asynchronously processable class data, the data may be asynchronously processed using an EDA mechanism. For example, asynchronously processing class data may include: short message EDA event. Illustratively, the SMS EDA event may be a debit card balance change SMS.

To ensure that EDA messages are not lost, the system creates an EDA event table that stores EDA event information that the system generates every day. The EDA event table is thus data-intensive and each transaction has an EDA table insertion operation, i.e. the EDA table insertion is time-consuming. In an embodiment, the short message EDA event may be transplanted to the secondary system, i.e. the secondary system is used to process the short message EDA event, while the short message EDA event is not registered in the primary system. Illustratively, the subscriber may include: a short message sender, a micro message sender and an APP. It can be understood that the user can be reminded of the balance change by sending a notification such as a balance change message or information to the user through the subscription.

In an embodiment, in the case that the type of data to be processed is newly added class data, the data processing policy is an EDA mechanism;

correspondingly, the data to be processed is processed according to the data processing strategy, which comprises the following steps: the main system converts the newly added class data into data in a second target file format; and sending the data in the second target file format to the target financial system so as to enable the target financial system to carry out data new addition.

In an embodiment, in the case that the data to be processed is newly added class data, an EDA mechanism may also be used to process the updated class data. For example, the newly added class data may be accounting pipeline data. In an embodiment, the new class data may be directly converted into data in the second target file format through the downshifting, that is, the data in the second target file format is sent to the target accounting system in a file manner instead of the original EDA mode, so as to perform data accounting.

In an embodiment, in a case that the secondary system accesses the primary system by using the DRDA technology, the method further includes: the auxiliary system sends a data query request to the main system; the main system adopts WLB technology to process the data inquiry request; the auxiliary system adopts a table-connected query mode to query the query class data in the main system; the subsystem adopts a static SQL mode to process the query data.

In the embodiment, the scheme is suitable for a scene of quick payment, and under the condition that the data to be processed is query data, as only two tables of a debit card account are required to be queried, namely the scene is single, the data table of the DB2 database can be queried in a continuous table query mode, so that the access times to the database in the main system are reduced. Meanwhile, a static SQL mode can be adopted to query the database in the main system, so that the query efficiency is improved. After the auxiliary system sends the data query request to the main system, in order to make the data query request be automatically balanced among a plurality of instances of the DB2 database, so as to solve the problem of overall performance reduction caused by unbalanced load, the WLB technology of IBM can be adopted, so as to achieve the purpose of automatic load balancing.

Illustratively, in one implementation, the analysis is performed with the transaction that is the greatest in transaction amount (e.g., debit card quick consumption transaction), and the steps of the transaction may generally include the steps of: institution checking, debit card checking, product checking, parameter querying, account processing, and running water registration. And then analyzing the data table accessed by the transaction according to the dimensions of reading, writing, data quantity and the like. Finally, obtaining that the parameter data is read-only data, the data quantity is small, and the data update is not frequent; debit card and account data are read-write type data (i.e. query type data in the above embodiments), and the data flow is huge and updated frequently; the stream data (i.e., the new class data in the above embodiment) is the new data for each transaction.

Based on the information, creating a copy on the Linux platform (namely, the second class system/subsystem) aiming at the parameter class data, and transplanting the access to the open platform (namely, the second class system); for inquiring debit card and account data, because the data flow is huge and the update is frequent, a copy cannot be created, and therefore the open platform accesses the database of the host DB2 through the DRDA technology; aiming at updating account data and adding stream data, the method still remains in a host, and reduces resource consumption through some optimization techniques.

FIG. 2 is a block diagram of a data processing system according to an embodiment of the present invention. As shown in fig. 2, the main system is a right-side P6 system, and the sub-system is a left-side P8 system. In an embodiment, parameter class data, query class data, update class data, and new class data are analyzed, respectively. The method comprises the following steps:

first, for the parameter class data, for example, the parameter class data includes: the mechanism parameter, the province parameter, the product parameter list and the pricing parameter list have the characteristics of low read-only property, little change, low timeliness requirement and the like, so that a data copy is created on an open platform and stored in an Oracle database (second class database).

In an embodiment, a data master copy synchronization mechanism may be employed in processing parameter class data. The data synchronization in the architecture is the data synchronization of the cross-platform heterogeneous database, and no ready-made data synchronization product can be directly used, namely a high-availability, controllable and adjustable data synchronization scheme in the scheme is provided.

After analyzing the update scene of the parameter class data, part of the parameter class data is allowed to be updated in the daytime, but the update frequency is low, and the update quantity is small; another part of the parameter class data only allows for end-of-day updates. Based on the characteristics, the incremental change file is generated after going more in the daytime based on the file synchronization mechanism; after the update is completed, a full change file is generated, and the file is transmitted from the main system to the sub-system through a file transmission engine in the system. After the secondary system receives the file, the multiple copies of the data are updated.

Secondly, for query class data, for example, the query class data includes: debit cards and their account data, the data volume is relatively large, the update is frequent, and the solution cannot be achieved by using a duplicate scheme. Thus, DB2 databases in the host system are accessed from the open platform using DRDA techniques. In the scenario where the DRDA accesses the host system, the host system uses the coprocessor to process, reducing the host processor pressure.

Although DRDA technology is a mature technology of IBM, in a practical transaction scenario, many optimizations have been performed. First, the number of accesses to the DRDA is reduced as much as possible. Because of the table structure design of the debit card and account, 6 accesses are required if accessed according to the master key sheet table. But the number of accesses can be reduced to 2 if complex linked list queries are used. Tests show that the scheme of 2 accesses is far superior to 6 accesses.

Second, DRDA processes SQL queries in two modes, dynamic SQL and static SQL. Through testing, the processing efficiency of the static SQL mode is doubled as that of the dynamic SQL mode, namely, the data is queried by adopting the static SQL mode.

And finally, processing the data query request by adopting the WLB technology of IBM, namely, automatically balancing the load among a plurality of instances of the DB2 by all the data requests, and solving the problem that the overall performance is reduced due to the fact that a single instance is crushed by unbalanced load.

Thirdly, processing the data to be processed in the auxiliary system by adopting a local cache and a cache synchronization mechanism. To further improve transaction performance, reduce transaction latency, reduce Oracle pressure, a local cache framework may be configured in the subsystem. The local cache framework is divided into the following parts:

cache configuration: the data table with the buffer function to be started is configured through XML, and the maximum record number of the buffer and the effective time of the buffer are configured. The effective time of the cache is set to 24 hours by default, considering that the parameters are updated substantially once per day.

Cache load and use: the framework dynamically cuts into the data access logic of the application. When the data table is accessed, the cache loading and the use are carried out according to the cache configuration, namely, if the cache is opened, the cache is accessed first, and if the cache does not exist, the database is accessed and then loaded into the cache.

Cache cleaning: in addition to having automatic cache cleaning, a global approach to cache cleaning is provided. And when the parameter copy database is updated in daytime or at the end of the day, a cache cleaning method of each AP is called, and the cache is cleaned, so that cache consistency is realized.

Fourth, the update class data and the new addition class data are aimed at. The update class data is data of balance change short message event, and the new class data is data of accounting running water event.

In the shortcut transaction, the balance change short message and accounting running water interact with the short message component and the accounting engine component through an EDA mechanism. To ensure that the EDA message must be sent successfully, EDA information is registered in the database before EDA occurs. Because EDA event table data volume is extremely huge, and each transaction has EDA table insertion operation, EDA table insertion is very time-consuming, and can bring about a lot of performance improvement after optimization. The optimization was performed by the following 2 ways:

in an embodiment, the short message EDA event is transplanted to the auxiliary system, and the main system does not register. Fig. 3 is a block diagram of a processing structure of a short message EDA event according to an embodiment of the present invention. As shown in fig. 3, above the dashed line is the P6 system, a prior art solution; below the dashed line is the P8 system, i.e. the technical solution in the embodiment of the invention. For example, assuming that the shortcut payment transaction is 5821 shortcut payment transaction, the P8 system acquires the target element in 5821 shortcut payment transaction (i.e. focus subscription judgment, focus post-processing data generation, etc. from the P6 system, then determines whether to record focus post-processing data into a transaction output message by combining a message and focus EDA switch flag in the TXP), namely, invokes 5821 transaction in the P8 system, loads focus post-processing data, and then writes focus post-processing information into an Oracle database; then, the P8EDA post-processing process reads data information in the Oracle database one by one, namely reads balance change attention information from the post-processing database, if the balance change attention information is sent to the open short message, balance change EDA events assembled into a multi-stroke message mode are sent to a message center for processing; if the balance change EDA event is sent to other subscribers, the balance change EDA event is assembled into a single-stroke message mode and is sent to a message center for processing; and finally sent to subscribers (e.g., subscriber 1, subscriber 2 … …, subscriber n) through message center processing.

In one embodiment, fig. 4 is a block diagram of a processing structure for accounting for a running water event according to an embodiment of the present invention. As shown in fig. 4, in the P6 system, accounting elements in the on-line transaction are recorded, and the accounting elements are written into an accounting information file a and an accounting information file B in odd-numbered batches and even-numbered batches, respectively; then downshifting is performed in batches, illustratively, three batches are respectively downshifted to an accounting processing timing batch 01, an accounting processing timing batch 02 and an accounting processing timing batch 03; then, accounting is carried out on the files after the lower file is put into account respectively; and finally, accounting and accounting are carried out on the files of the final batch in the day.

In order to solve the problem, the accounting flow table can be split into a plurality of sub-tables, and the round trip flow is switched, so that the lower file can not influence the access of the data table. For example, the downshifting may be performed at a time point, such as by archiving the accounting information file in the three time periods of early, middle, and late.

In an embodiment, a centralized and distributed combined framework is adopted, so that the lateral expansion is facilitated, for example, as the transaction amount increases year by year, a second-class system (subsystem) server can be laterally expanded, and enough service capability can be provided; meanwhile, the transaction delay is reduced, and the transaction delay is greatly reduced by using a caching technology; meanwhile, the cost is reduced, the cost of the server of the second class system is far lower than the hardware cost of the main system, and the hardware cost is greatly reduced under the condition of providing the same service capability; and an open framework is adopted, so that advanced technology can be introduced conveniently, and better service is provided.

FIG. 5 is a block diagram of a data processing apparatus according to an embodiment of the present invention, which is suitable for use in data processing using a centralized and distributed fusion framework, and which may be implemented in hardware/software and may be generally integrated in a data processing system. As shown in fig. 5, the apparatus includes: a first determination module 210, a second determination module 220, and a processing module 230.

Wherein, the first determining module 210 is configured to determine a type of the received data to be processed;

a second determining module 220, configured to determine a corresponding data processing policy according to a type of the data to be processed;

the processing module 230 is configured to process the data to be processed according to the data processing policy.

According to the technical scheme, the data to be processed is processed by adopting the distributed and centralized fusion framework, so that the dependence on the IBM mainframe is reduced, the cost overhead of hardware resources is reduced, and the computing capacity of the system is improved.

In one embodiment, the type of data to be processed includes one of: parameter class data, query class data, update class data, and newly add class data.

correspondingly, the data to be processed is processed according to the data processing strategy, which comprises the following steps:

the auxiliary system receives intermediate data synchronously updated by the main system, wherein the intermediate data is data in a first target file format obtained by archiving data to be processed by the main system;

and the subsystem analyzes the intermediate data into data in a corresponding table structure file format and accesses the data in the table structure file format.

In an embodiment, in the case that the type of the data to be processed is query class data, the data processing policy is a Distributed Relational Database Architecture (DRDA) technology access;

receiving a data access request of a subsystem;

determining a corresponding access interface according to the data access request;

and determining whether to process the data to be processed by adopting a coprocessor of the main system according to the access interface.

In an embodiment, in case the type of data to be processed is update class data, the data processing policy is an event driven architecture EDA mechanism;

the auxiliary system acquires a target element of update type data in the main system;

synchronizing and storing the target elements to a target database of the subsystem itself;

and under the condition that the EDA sending thread is detected, sending the target elements in the target database to a target message center so as to enable the subscriber to update the data.

the main system converts the newly added class data into data in a second target file format;

and sending the data in the second target file format to the target financial system so as to enable the target financial system to carry out data new addition.

In an embodiment, in a case that the secondary system accesses the primary system by using the DRDA technology, the method further includes:

the auxiliary system sends a data query request to the main system;

the main system adopts WLB technology to process the data inquiry request;

the auxiliary system adopts a table-connected query mode to query the query class data in the main system;

the subsystem adopts a static SQL mode to process the query data.

The data processing device can execute the data processing method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

FIG. 6 is a schematic diagram of a hardware architecture of a data processing system according to an embodiment of the present invention. As shown in fig. 6, a data processing system provided in an embodiment of the present invention includes: a primary system 310 and a secondary system 320; the main system 310 adopts a first type system and a first type database; the subsystem 320 adopts a second type system and a second type database; the host system 310 includes: a memory 3101, a main processor 3102, and one or more coprocessors 3103; the subsystem 320 includes: memory 3201, and one or more processors 3202. The coprocessor 3103 and processor 3202 in the data processing system may be one or more, one coprocessor 3103 and one processor 3202 are illustrated in fig. 6, the memory 3101 in the data processing system, the main processor 3102 and the coprocessor 3103 may be connected by a bus or other means, and the memory 3201 and the processor 3202 may be connected by a bus or other means, illustrated in fig. 6 as a bus connection.

The memory 3101 and the memory 3201 in the data processing system are used as a computer readable storage medium, and may be used to store one or more programs, which may be software programs, computer executable programs, and modules, such as program instructions/modules corresponding to the data processing method provided in the embodiment of the present invention (for example, the modules in the data processing apparatus shown in fig. 5, including the first determining module 210, the second determining module 220, and the processing module 230). The main processor 3102 and the coprocessor 3103 execute various functional applications of the device and data processing by running software programs, instructions and modules stored in the memory 3101, that is, implement the data processing method in the above-described method embodiment; the memory 3201 executes various functional applications of the apparatus and data processing by running software programs, instructions and modules stored in the processor 3202, that is, implements the data processing method in the above-described method embodiment.

Memory 3101 and memory 3201 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, at least one application program required for functions; the storage data area may store data created according to the use of a device configured in the device, and the like. Further, memory 3101 and memory 3201 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some examples, memory 3101 may further include memory located remotely from main processor 3102 and co-processor 3103, or memory 3201 may further include memory located remotely from processor 3202, which may be connected to a device configured in the device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

In one embodiment, a data processing system is provided, including a primary system 310 and a secondary system 320; the host system 310 includes: a memory 3101, a main processor 3102, and one or more coprocessors 3103; a subsystem 320, comprising: memory 3201, and one or more processors 3202. The memory 3101 and the memory 3201 store computer programs, the main processor 3102, and one or more coprocessors 3103, or the processor 3202 when executing the computer programs performs the steps of:

determining the type of the received data to be processed; determining a corresponding data processing strategy according to the type of the data to be processed; and processing the data to be processed according to the data processing strategy.

The data processing system can execute the data processing method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

The embodiment of the invention also provides a computer readable storage medium, on which a computer program is stored, which when executed by a processor, implements the data processing method provided by the embodiment of the invention, and the method includes: determining the type of the received data to be processed; determining a corresponding data processing strategy according to the type of the data to be processed; and processing the data to be processed according to the data processing strategy.

The computer storage media of embodiments of the invention may take the form of any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations of the present invention may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).

Note that the above is only a preferred embodiment of the present invention and the technical principle applied. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.

Claims

1. A method of data processing, comprising:

determining the type of the received data to be processed;

processing the data to be processed according to the data processing strategy;

in the case that the type of the data to be processed is parameter data, the data processing strategy is a data primary and secondary synchronization mechanism of a cross-platform heterogeneous database;

correspondingly, the processing the data to be processed according to the data processing policy includes:

the method comprises the steps that a secondary system receives intermediate data synchronously updated by a primary system, wherein the intermediate data is data in a first target file format obtained by archiving data to be processed by the primary system;

the subsystem analyzes the intermediate data into data in a corresponding table structure file format and accesses the data in the table structure file format;

the main system is a centralized system, and the auxiliary system is a distributed system.

2. The method of claim 1, wherein the type of data to be processed comprises one of: parameter class data, query class data, update class data, and newly add class data.

3. The method according to claim 2, wherein in case the type of data to be processed is query class data, the data processing policy is a distributed relational database architecture, DRDA, technology access;

receiving a data access request of a subsystem;

and determining whether to process the data to be processed by adopting a coprocessor of a main system according to the access interface.

4. The method according to claim 2, wherein in case the type of data to be processed is update class data, the data processing strategy is an event driven architecture, EDA, mechanism;

5. The method according to claim 2, wherein in case the type of data to be processed is an add-on class data, the data processing strategy is an EDA mechanism;

and sending the data in the second target file format to a target financial system so as to enable the target financial system to carry out data new addition.

6. A method according to claim 3, wherein in case the secondary system accesses the primary system using DRDA technology, further comprising:

the auxiliary system sends a data query request to the main system;

the master system adopts a WLB technology to process the data query request;

and the subsystem adopts a static SQL mode to process the query data.

7. A data processing apparatus, comprising:

the processing module is used for processing the data to be processed according to the data processing strategy;

8. A data processing system, comprising: a main system and a sub-system; the main system adopts a first type system and a first type database; the subsystem adopts a second class system and a second class database;

the subsystem includes: a memory, and one or more processors;

the memory is used for storing one or more programs;

when executed by the one or more processors, causes the one or more processors to implement the data processing method of any of claims 1-6.

9. A computer-readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements a data processing method as claimed in any one of claims 1-6.