CN115640345A - Data processing method, device, system, equipment, storage medium and product - Google Patents
Data processing method, device, system, equipment, storage medium and product Download PDFInfo
- Publication number
- CN115640345A CN115640345A CN202211185399.6A CN202211185399A CN115640345A CN 115640345 A CN115640345 A CN 115640345A CN 202211185399 A CN202211185399 A CN 202211185399A CN 115640345 A CN115640345 A CN 115640345A
- Authority
- CN
- China
- Prior art keywords
- upstream
- data processing
- data
- adaptation layer
- adaptation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing method, a device, a system, equipment, a storage medium and a product. The invention relates to the technical field of computers. The method is applied to a data processing system and comprises the following steps: the method comprises the steps of responding to triggering of an upstream data processing task, determining at least one upstream system corresponding to the upstream data processing task, processing source data in the at least one upstream system into standard interface data based on an adaptation layer corresponding to the at least one upstream system in the data processing system, wherein the standard interface data meet business logic requirements corresponding to the upstream data processing task in the data processing system, and processing the standard interface data based on a preset standard interface processing strategy corresponding to the upstream data processing task to obtain task result data corresponding to the upstream data processing task. The technical scheme of the invention can quickly adapt to different upstream systems and reduce the code development cost when the upstream systems change.
Description
Technical Field
Embodiments of the present invention relate to the field of computer technologies, and in particular, to a data processing method, apparatus, system, device, storage medium, and product.
Background
The interaction between the systems can be divided into two modes, namely a transaction line interaction mode and a data line interaction mode. The interactive mode of the data line has a form that the system is used as a downstream and is accessed to the data of the upstream for data processing; the system also serves as an upstream system and provides data to a downstream system for use.
The existing upstream and downstream data line interaction mode is as follows: the method comprises the steps of docking with an upstream system, knowing a table model of the upstream system, determining the meaning and code value of each data field, accessing upstream data, processing the upstream data according to the requirement of the system, storing a processing result in the system, docking with a downstream system, determining the data content and definition required by the downstream, processing the data of the system according to the downstream requirement, generating the data content meeting the downstream requirement, and providing the data content to the downstream system. The data line interaction mode coupling between the upstream system and the downstream system is serious, and after any side is changed due to the addition or modification of the system, huge modification cost is brought to the system. The code needs to be developed again, and the cost is high.
Disclosure of Invention
The embodiment of the invention provides a data processing method, a device, a system, equipment, a storage medium and a product, which can optimize a data interaction scheme of an upstream system and a downstream system.
In a first aspect, an embodiment of the present invention provides a data processing method, applied to a data processing system, including:
in response to the upstream data processing task being triggered, determining at least one upstream system corresponding to the upstream data processing task;
processing source data in at least one upstream system into standard interface data based on an adaptation layer corresponding to at least one upstream system in a data processing system, wherein the standard interface data meet business logic requirements corresponding to upstream data processing tasks in the data processing system;
and processing the standard interface data based on a preset standard interface processing strategy corresponding to the upstream data processing task to obtain task result data corresponding to the upstream data processing task.
In a second aspect, an embodiment of the present invention further provides a data processing apparatus, where the apparatus includes:
the upstream system determining module is used for determining at least one upstream system corresponding to the upstream data processing task in response to the upstream data processing task being triggered;
the source data processing module is used for processing the source data in at least one upstream system into standard interface data based on an adaptation layer corresponding to at least one upstream system in the data processing system, wherein the standard interface data meet the business logic requirement corresponding to the upstream data processing task in the data processing system;
and the interface data processing module is used for processing the standard interface data based on a preset standard interface processing strategy corresponding to the upstream data processing task so as to obtain task result data corresponding to the upstream data processing task.
In a third aspect, an embodiment of the present invention further provides a data processing system, where the system is configured to implement the data processing method according to any one of the embodiments of the present invention.
In a fourth aspect, an embodiment of the present invention further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor executes the computer program to implement the data processing method according to any one of the embodiments of the present invention.
In a fifth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the data processing method according to any one of the embodiments of the present invention.
In a sixth aspect, the present invention further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the computer program implements the data processing method according to any one of the embodiments of the present invention.
According to the data processing scheme provided by the embodiment of the invention, the data processing system responds to the triggering of the upstream data processing task, at least one upstream system corresponding to the upstream data processing task is determined, based on the adaptation layer corresponding to the at least one upstream system in the data processing system, the source data in the at least one upstream system is processed into the standard interface data, based on the preset standard interface processing strategy corresponding to the upstream data processing task, the standard interface data are processed to obtain the task result data corresponding to the upstream data processing task, by introducing the adaptation layer and the standard interface data, when the upstream system is connected, the data processing logic with data interaction in the system is divided into two parts, one part is adapted to the upstream system, and the other part is unrelated to the upstream system, so that the data processing system has the capability of being capable of rapidly adapting to different upstream systems, and the code development cost when the upstream system changes is reduced.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a data processing process according to an embodiment of the present invention;
FIG. 3 is a flow chart of another data processing method according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not to be construed as limiting the invention. It should be further noted that, for the convenience of description, only some structures related to the present invention are shown in the drawings, not all of them.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined or explained in subsequent figures. Meanwhile, in the description of the present invention, the terms "first", "second", and the like are used only for distinguishing the description, and are not construed as indicating or implying relative importance. According to the technical scheme, the data acquisition, storage, use, processing and the like meet the relevant regulations of national laws and regulations.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention, where the data processing method is applicable to a situation where an upstream system and a downstream system in a data processing system perform data interaction, and the method may be executed by a data processing apparatus, where the apparatus may be implemented in a form of software and/or hardware, and optionally implemented by an electronic device such as a data processing device, where the data processing device may be a mobile terminal such as a mobile phone or a tablet Computer, or may be a device such as a Personal Computer (PC) end or a server.
As shown in fig. 1, the method includes:
Illustratively, in a data line interaction process, involving an upstream system as well as a downstream system, given the terms "upstream" and "downstream," an analogy can be made with a river. If a message (data) is placed in a river, it will flow from upstream (initiator) to downstream (receiver). It should be noted that the upstream system and the downstream system are opposite upstream and downstream systems, and are generally systems composed of a plurality of services invoked with each other, and are not fixed upstream systems or downstream systems, the upstream and downstream are determined by the direction of data flow, and the manner of data flow in the upstream or downstream is determined by the final processor. In the embodiment of the present invention, the data processing system may be used as a downstream system or an upstream system.
The upstream data processing task may be understood as a task for performing data processing on upstream data (also referred to as source data) from an upstream system, and specifically may be understood as a task corresponding to the whole data processing process of obtaining data from the upstream system and processing target data (task result data) required by the cost system, and data processing logic corresponding to the task, requirements for data processing results, and the like may be set according to actual business requirements of the data processing system. After the upstream data processing task is triggered, the current identity of the data processing system may be determined to be the downstream system.
Illustratively, the upstream data processing task is triggered, and the corresponding upstream system is determined according to the upstream data processing task, where the number of the upstream systems may be one or more, and is not limited specifically, and if the number of the upstream systems is multiple, batch interactive processing of data may be implemented, and different interactive data are processed according to different upstream systems.
In the related art, when a downstream system processes upstream data, a whole set of codes needs to be developed according to a complete data processing logic, that is, an upstream and downstream data line interaction mode is completely coupled with data rules of the upstream and downstream systems, and when the upstream system changes, the codes need to be redeveloped to meet requirements of various aspects such as a changed system data format.
In the embodiment of the invention, the upstream data processing logic in the downstream system is divided into two stages, the first stage is adapted to the upstream system, and the second stage is independent of the upstream system and depends on the service logic requirement of the downstream system. The first stage is realized by an adaptation layer, and the second stage is realized by a preset standard interface processing strategy.
Illustratively, the adaptation layer is configured in the downstream system, corresponding to at least one upstream system, for performing data conversion to convert the source data in the upstream system into standard interface data. Optionally, in a case that the upstream data processing task corresponds to multiple upstream systems, at least two upstream systems exist and correspond to different adaptation layers. The upstream data processing task may correspond to one or more upstream systems, when the upstream data processing task corresponds to two or more upstream systems, at least two upstream systems corresponding to different adaptation layers exist, the adaptation layers may correspond to the upstream systems one to one, or a plurality of upstream systems correspond to one adaptation layer, that is, the adaptation layers may be individually configured or multiplexed according to characteristics of the upstream systems.
For example, the data processing system may be configured with a standard interface, and the standard interface may be understood as a series of lists or fields abstracted from the own system business logic (i.e. business logic corresponding to an upstream data processing task) of the data processing system, which represent information that the own system of the data processing system depends on the upstream. The standard interface data may be understood as the table or the field filled with specific data content and meeting the requirement of the standard interface, so that the standard interface data meets the service logic requirement corresponding to the upstream data processing task in the data processing system, that is, the upstream data in the upstream system is transmitted to the downstream system and processed into the table or the field required by the standard interface through the adaptation layer, optionally, the source data is another name of the upstream data, and the standard interface data is general data that can be identified by the standard interface after the source data is processed by the adaptation layer.
Illustratively, the data processing system includes an adaptation layer corresponding to each upstream system, and the adaptation layer processes, according to different access rules (such as acquired data, data type, data acquisition method, and data acquisition source) of different upstream systems, the source data transmitted in each upstream system into general standard interface data that can be recognized and processed by a standard interface by using adaptation layer processing logic.
And 103, processing the standard interface data based on a preset standard interface processing strategy corresponding to the upstream data processing task to obtain task result data corresponding to the upstream data processing task.
If an upstream system is added, an adaptation layer can be added to correspond to the newly added upstream system, and similarly, the upstream system is reduced, and the adaptation layer corresponding to the newly added upstream system can be reduced. The standard interface does not need to be changed, namely, the code development amount is carried out on an adaptation layer instead of the whole processing logic during data interaction; and the second stage of data processing logic is the business processing logic of the system, and standard interface data are processed into the database of the system according to a preset standard interface strategy corresponding to the upstream data processing task. This data processing is relatively stable logic, which does not change due to changes in the external system (e.g., replacement of the upstream system or changes in the rules of the upstream system).
The preset standard interface processing policy corresponding to the upstream data processing task may be accumulation or other processing of standard interface data corresponding to each upstream system, and the specific processing policy is not limited.
For example, fig. 2 is a schematic diagram of a data processing process provided by an embodiment of the present invention, as shown in fig. 2, the data processing system may be a credit flow system, which needs to access a core system (upstream system) to synchronously release loan information such as repayment and the like, and is used to process contract accounting information 29 in the system, that is, the upstream data processing task is to process contract accounting information. Illustratively, the upstream data processing task may correspond to one or more upstream systems, for example, may correspond to three upstream systems and one downstream system, where the three upstream systems are system a20, system B21, and system C22, respectively, the downstream system is system D23 (i.e. the data processing system in the embodiment of the present invention), and in the system D23, three system a adaptation layers 24, three system B adaptation layers 25, and three system C adaptation layers 26, which correspond to system a21, system B22, and system C23, respectively, and one standard interface are included. The standard interface may form different adaptations depending on different upstream data processing tasks. In the credit flow system, the standard interface is a loan account standard interface 27, and the loan account standard interface 27 defines a field of accumulated loan amount, which represents the interactive definition of the system and the upstream system, that is, the system needs to provide the accumulated loan amount upstream or process the information of the accumulated loan amount. And processing the accumulated loan amount of the standard interface loan account level into the accumulated loan amount of the contract level of the system by using the business processing logic of the system of the downstream system to obtain target data of the system, namely task result data corresponding to the upstream data processing task.
Specifically, the upstream data in the system a20 may be the total loan amount, and in the data processing task, the field-to-field direct mapping is performed on the upstream data in the system a adaptation layer 24, and the upstream data is processed into a field of the accumulated loan amount to the loan account standard interface 27; the upstream data in the system B21 may be at least two segments of data, for example, a loan amount 1 and a loan amount 2, in this data processing task, two fields of the upstream data are accumulated in the adaptation layer 25 of the system B, which is specifically expressed as the loan amount 1+ the loan amount 2, and the upstream data is processed into a field of the accumulated loan amount to the loan account standard interface 27; the upstream data in the system C22 may be intermittent data information, for example, may be running water information of loan amount, and in this data processing task, the required running water is fetched and processed in the system C adaptation layer 26, and the running water in the upstream data is filtered into effective running water and accumulated to obtain a field of the standard interface of the loan account, that is, the accumulated loan amount, which is processed to the standard interface 27 of the loan account. After the standard interface data of the loan accounts of each upstream system is obtained, the amount is accumulated and summarized 28 to obtain contract financial information 29 according to a plurality of loan accounts corresponding to one contract, namely, the accumulated loan amount of the standard interface loan account level is processed into the accumulated loan amount of the contract level of the system, and the task result data corresponding to the upstream data processing task is obtained through the processing of two stages of an adaptation layer and a preset standard interface processing strategy according to the upstream data of each upstream system.
The data processing method provided by the embodiment of the invention is applied to a data processing system, responds to the triggering of an upstream data processing task, determines at least one upstream system corresponding to the upstream data processing task, processes source data in the at least one upstream system into standard interface data based on an adaptation layer corresponding to the at least one upstream system in the data processing system, and processes the standard interface data based on a preset standard interface processing strategy corresponding to the upstream data processing task to obtain task result data corresponding to the upstream data processing task. By introducing the adaptation layer and the standard interface data, when an upstream system is butted, the data processing logic with data interaction in the system is divided into two parts, one part is adapted to the upstream system, and the other part is irrelevant to the upstream system, so that decoupling is realized, the data processing system has the capability of being rapidly adapted to different upstream systems, and the code development cost when the upstream system changes is reduced.
Optionally, in some embodiments, the data processing method may further include: and responding to the triggering of an upstream system change event, acquiring change operation data, and performing change processing on an adaptation layer in the data processing system according to the change operation data. The advantage of this arrangement is that the adaptation layer in the data processing system can be dynamically adjusted according to the change of the upstream system, thereby quickly implementing adaptation and reducing the code development cost.
For example, the upstream system change event may be an event corresponding to an upstream system modification or a new upstream system, and may be denoted as an original upstream system modification event and a new upstream system event, respectively. And after the upstream system change event is triggered, acquiring change operation data according to the upstream system change event, wherein the change operation data can comprise data which is generated after a user inputs a change operation and is related to the adaptation layer.
Optionally, in a case that the upstream system change event is an original upstream system modification event, the change operation data includes adaptation layer modification information. Wherein, the changing the adaptation layer in the data processing system according to the change operation data includes: and modifying the adaptation layer corresponding to the original upstream system in the target indicated by the original upstream system modification event according to the adaptation layer modification information. The method has the advantages that when the original existing upstream system changes, the adaptation layer corresponding to the changed original upstream system, which exists in the data processing system, can be modified, and the change of the original upstream system can be adapted quickly.
For example, taking the system B21 as an example, if the field of the accumulated deposit amount is added to the system B21 due to the upgrade of the system B21, the system B adaptation layer 25 may be adjusted to directly map the fields one to one.
Fig. 3 is a flowchart of another data processing method according to an embodiment of the present invention, where the embodiment of the present disclosure performs optimization based on various alternatives in the above embodiments, adds related steps for changing an upstream system event, and takes an example that the upstream system event is a newly added upstream system event.
Illustratively, in a case where the upstream system change event is a newly added upstream system event, the change operation data includes adaptation layer configuration information; wherein, the changing the adaptation layer in the data processing system according to the change operation data includes: and constructing an adaptation layer corresponding to the target new upstream system indicated by the newly added upstream system event according to the configuration information of the adaptation layer. The method has the advantages that after the newly-added upstream system is added, the adaptation layer corresponding to the newly-added upstream system can be quickly and accurately constructed according to the changed operation data, the code development cost is reduced, the efficiency of the data processing system for connecting the new upstream system is improved, and the availability and the service processing capacity of the data processing system are improved.
Further, in response to an event of a newly added upstream system being triggered, acquiring adaptation layer configuration information, including: in response to a newly added upstream system event being triggered, providing an adaptation layer configuration template to a user; receiving configuration content input by a user based on the adaptation layer configuration template; and determining adaptation layer configuration information according to the adaptation layer configuration template and the configuration content. The advantage of this arrangement is that by providing the template, adaptation layer configuration efficiency and construction efficiency are improved.
Illustratively, the method may include:
For example, before executing this step, the data processing system may perform normal processing on the upstream data processing task, that is, process the source data in the existing upstream system into standard interface data based on the adaptation layer corresponding to the existing upstream system, and then process the standard interface data based on the corresponding preset standard interface processing policy to obtain corresponding task result data.
With the change of the service requirement, a new upstream system may be added, and the new upstream system may be adapted quickly by constructing an adaptation layer corresponding to the new upstream system.
For example, after a new upstream system event is triggered, an adaptation layer configuration template is provided to the user, and the adaptation layer configuration template may include configuration items required for building an adaptation layer, such as a job identifier, a processing target table, a processing target field, a mapping rule, a source table, an association rule between source tables, and the like. Optionally, the adaptation layer configuration template may be displayed in a table form, such as an Excel table, and the configuration items are displayed in a header.
And step 320, receiving configuration contents input by a user based on the adaptation layer configuration template.
For example, after the adaptation layer configuration template is presented to the user, the user may input specific configuration contents for each configuration item, for example, fill in the corresponding contents of each header in the Excel table.
Illustratively, the configuration content is filled into the corresponding position in the adaptation layer configuration template to obtain the adaptation layer configuration information.
And 340, constructing an adaptation layer corresponding to the target new upstream system indicated by the newly added upstream system event according to the configuration information of the adaptation layer.
Optionally, the adaptation layer is implemented by Structured Query Language (SQL) code. For example, an adaptation layer may correspond to a piece of SQL code. When the adaptation layer is constructed, the corresponding SQL code can be automatically generated according to the configuration information of the adaptation layer, so that the adaptation layer corresponding to the target new upstream system indicated by the newly-added upstream system event is constructed.
Optionally, the adaptation layer includes a first adaptation logic and/or a second adaptation logic, where the first adaptation logic includes a mapping rule at a field level between a source table in the source data and a standard interface table in the standard interface data, and the second adaptation logic includes an association rule of at least two source tables in the source data. This has the advantage that different types of upstream systems can be more fully adapted.
For example, different adaptation layer configuration templates may be provided to a user for different adaptation logics, a first adaptation logic may correspond to the first adaptation layer configuration template, and a second adaptation logic may correspond to the second adaptation layer configuration template.
Optionally, the adaptation layer configuration template may be embodied in a configuration table, and taking an example that the adaptation layer of the target new upstream system includes the first adaptation logic and the second adaptation logic at the same time, the configuration table includes configuration 1-1 and configuration 1-2.
Illustratively, the configuration table is shown in Table 1 below:
the configuration table 1 comprises a configuration 1-1 and a configuration 1-2, the configuration 1-1 represents a field-level mapping rule, and represents how to use one or more fields of a source table to map to fields of a standard interface table through a certain mapping rule, the standard interface table can be understood as a table required by a standard interface and can also be called a processing target table, and the configuration 1-1 comprises a job identifier, a standard interface table, a processing target field, a mapping rule, a source table and a source table alias to form a first adaptation layer configuration template. The content filled in the configuration 1-1 can be understood as the configuration content input by the user. When the processing target field is COLUMN-1, the mapping rule is a.COLUMN-A, and the mapping is direct mapping; when the processing target field is COLUMN-2, the mapping rule is decode (a. COLUMN-B,1, A,2, B), if the upstream value is 1, A is returned, and if the upstream value is 2, B is returned, namely, the presentation forms of the data are mutually converted; and when the processing target field is COLUMN-3, the mapping rule is b.COLUMN-C + b.COLUMN-D, namely, the fields are added up accumulatively. Configurations 1-2 show how the various source tables are associated. And receiving a plurality of tables contained in one upstream system or receiving each configuration table from different upstream systems at the adaptation layer, and associating each table through configuration 1-2, wherein the configuration 1-2 comprises job identification, a source table, source table alias, association relation, association condition and filter condition to form a second adaptation layer configuration template. The content filled in the configuration 1-2 can be understood as the configuration content input by the user. As shown in the above TABLE, the association condition is a.xx = b.xx, the association relationship may be a main TABLE, an internal association or a left association, the internal association TABLE-a INNER JOIN TABLE-B, and the result is the intersection of TABLE-a and TABLE-B; the LEFT correlation TABLE-A LEFT JOIN TABLE-B, the result is the intersection of all data in TABLE-A and TABLE-B. The purpose of configuration 1-2 is to associate the source tables together, so as to facilitate data interaction, for example, when the value of a field in the a table needs to be taken out from the B table.
Optionally, constructing, according to the adaptation layer configuration information, an adaptation layer corresponding to a target new upstream system indicated by the newly added upstream system event includes: determining a target database type corresponding to the data processing system, wherein optionally, the target database type may be a database such as MySQL, oracle or SQL Server; and generating an SQL code matched with the type of the target database according to the configuration information of the adaptation layer so as to construct the adaptation layer corresponding to the target new upstream system indicated by the newly added upstream system event. The advantage of this arrangement is that SQL code conforming to the database type can be automatically generated.
Illustratively, in the oracle database, according to the configuration template of the adaptation layer shown in configuration table 1, the automatically generated sql statement is as follows:
Insert into TABLE-1(COLUMN-1,COLUMN-2,COLUMN-3)
Select a.COLUMN-A,decode(a.COLUMN-B,1,A,2,B),
b.COLUMN-C+b.COLUMN-D
From TABLE-A a
Left join TABLE-B b on a.xx＝b.xx
For example, after an upstream system is newly added, the upstream system corresponding to the upstream data processing task may be changed, for example, the upstream system corresponding to the currently triggered upstream data processing task is newly added, and if the upstream data processing task has been triggered and the source data in the original upstream system has been processed into the standard interface data, at least one upstream system corresponding to the upstream data processing task determined here may be the target new upstream system. Optionally, if the source data in the original upstream system has changed, or in order to ensure the accuracy of data processing, the at least one upstream system corresponding to the upstream data processing task determined herein may include the original upstream system and the target new upstream system.
And step 360, processing the source data in the at least one upstream system into standard interface data based on an adaptation layer corresponding to the at least one upstream system in the data processing system.
Illustratively, for the target new upstream system, the source data in the target new upstream system is processed into standard interface data based on the newly constructed adaptation layer in step 340.
Illustratively, the interface processing policy is based on a preset standard corresponding to the upstream data processing task. And processing the standard interface data corresponding to the original upstream system and the standard interface data corresponding to the target new upstream system to obtain new task result data corresponding to the upstream data processing task.
The data processing method provided by the embodiment of the invention is applied to a data processing system, the data processing system responds to the triggering of a newly-added upstream system event, provides an adaptation layer configuration template for a user, receives configuration content input by the user based on the adaptation layer configuration template, determines adaptation layer configuration information according to the adaptation layer configuration template and the configuration content, constructs an adaptation layer corresponding to a target new upstream system indicated by the newly-added upstream system event, responds to the triggering of the upstream data processing task, determines at least one upstream system corresponding to the upstream data processing task, processes source data in at least one upstream system into standard interface data based on the adaptation layer corresponding to at least one upstream system in the data processing system, and processes the standard interface data based on a preset standard interface processing strategy corresponding to the upstream data processing task to obtain task result data corresponding to the upstream data processing task. By adopting the technical scheme, when an upstream system needs to be newly added, the corresponding adaptation layer can be quickly generated based on the adaptation layer configuration template provided by the system and the configuration content input by the user, so that the code development cost can be effectively reduced, the efficiency of the data processing system for connecting the new upstream system is improved, and the availability and the service processing capacity of the data processing system are improved.
In some embodiments, in the case that the upstream system change event is an original upstream system modification event, the adaptation logic may adjust by modifying configuration information of the adaptation layer, and the adaptation layer modification information may be information that modifies the adaptation layer configuration information. For example, the adaptation layer configuration information formed based on the adaptation layer configuration template may be modified to adjust, for example, modify configuration contents of configuration items in a configuration table, for example, modify a certain mapping rule to obtain new adaptation layer configuration information, and reconstruct an adaptation layer corresponding to a target original upstream system indicated by an original upstream system modification event according to the new adaptation layer configuration information, so as to modify the adaptation layer. Specifically, the new adaptation layer configuration information can be automatically converted into the corresponding SQL code according to the database type of the system.
Optionally, in some embodiments, the method may further include: responding to the triggering of a downstream data processing task, and processing data to be issued in the data processing system based on a preset unified interface processing strategy corresponding to the downstream data processing task to obtain interface data to be issued; determining at least one downstream system corresponding to the downstream data processing task, and processing the interface data to be issued into adaptation data to be issued based on an adaptation layer corresponding to the at least one downstream system in the data processing system, wherein the adaptation data to be issued meets the business logic requirement corresponding to the at least one downstream system; and sending the adaptation data to be issued to a corresponding downstream system. The method has the advantages that by introducing the interface data to be issued and the adaptation layer, when the downstream system is butted, the data processing logic with data interaction in the system is divided into two parts, one part is irrelevant to the downstream system, and the other part is adapted to the downstream system, so that decoupling is realized, the data processing system has the capability of being rapidly adapted to different downstream systems, and the code development cost when the downstream system is changed is reduced.
For example, the downstream data processing task may be understood as a task for processing data that needs to be delivered to a downstream system, and specifically may be understood as a task corresponding to the entire data processing process of processing data to be delivered in the system into adaptive data (task result data) to be delivered required by the downstream system, where a result of the downstream data processing task is used as an input of the downstream task.
Illustratively, the downstream data processing task is triggered, and the downstream system corresponding to the downstream data processing task is determined according to the downstream data processing task, where the downstream system may be one or more, and is not limited specifically, and if the downstream system is multiple, batch interactive processing of data may be implemented, and different interactive data are processed according to different downstream systems.
Illustratively, the data processing system is matched with one or more downstream systems, the data processing system comprises an adaptation layer corresponding to each downstream system, data to be transmitted is processed into standard interface data to be transmitted based on a preset unified interface processing strategy corresponding to a downstream data processing task, then the interface data to be transmitted is processed into the adaptation data to be transmitted based on the adaptation layer, that is, different downstream systems can receive the adaptation data to be transmitted matched with self business logic requirements through the processing of each adaptation layer.
In some embodiments, where the downstream data processing task corresponds to multiple downstream systems, there are at least two downstream systems corresponding to different adaptation layers.
In some embodiments, further comprising: acquiring second change operation data (which may be referred to as first change operation data in the foregoing to facilitate distinction from the related content of the upstream system change event in the foregoing) in response to the downstream system change event being triggered; and changing the adaptation layer in the data processing system according to the second change operation data.
In some embodiments, in a case that the downstream system change event is an original downstream system modification event, the second change operation data includes adaptation layer modification information; wherein, the changing the adaptation layer in the data processing system according to the second change operation data includes: and modifying the adaptation layer corresponding to the target original downstream system indicated by the original downstream system modification event according to adaptation layer modification information in the second change operation data.
In some embodiments, in a case that the downstream system change event is a newly added downstream system event, the second change operation data includes adaptation layer configuration information; wherein, the changing the adaptation layer in the data processing system according to the second change operation data includes: and constructing an adaptation layer corresponding to a target new downstream system indicated by the newly added downstream system event according to the adaptation layer configuration information in the second change operation data.
In some embodiments, in response to the new downstream system event being triggered, obtaining adaptation layer configuration information in the second change operation data includes: in response to a newly added downstream system event being triggered, providing a downstream adaptation layer configuration template to a user; receiving downstream configuration content input by a user based on the downstream adaptation layer configuration template; and determining adaptation layer configuration information in second change operation data according to the downstream adaptation layer configuration template and the downstream configuration content.
In some embodiments, the constructing, according to the adaptation layer configuration information in the second change operation data, the adaptation layer corresponding to the target new downstream system indicated by the newly added downstream system event includes: determining a second target database type corresponding to the target new downstream system; and generating an SQL code matched with the second target database type according to the configuration information of the adaptation layer in the second change operation data so as to construct the adaptation layer corresponding to the target new downstream system indicated by the newly added downstream system event.
Fig. 4 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention, as shown in fig. 4, the apparatus includes:
an upstream system determining module 41, configured to determine, in response to an upstream data processing task being triggered, at least one upstream system corresponding to the upstream data processing task;
a source data processing module 42, configured to process, based on an adaptation layer corresponding to the at least one upstream system in the data processing system, source data in the at least one upstream system into standard interface data, where the standard interface data meets a business logic requirement corresponding to the upstream data processing task in the data processing system;
and an interface data processing module 43, configured to process the standard interface data based on a preset standard interface processing policy corresponding to the upstream data processing task, so as to obtain task result data corresponding to the upstream data processing task.
According to the data processing device provided by the embodiment of the invention, by introducing the adaptation layer and the standard interface data, when an upstream system is butted, the data processing logic with data interaction in the system is divided into two parts, one part is adapted to the upstream system, and the other part is irrelevant to the upstream system, so that decoupling is realized, the data processing system has the capability of rapidly adapting to different upstream systems, and the code development cost when the upstream system changes is reduced.
Optionally, in a case that the upstream data processing task corresponds to multiple upstream systems, at least two upstream systems exist and correspond to different adaptation layers.
Optionally, the apparatus further comprises:
the change data acquisition module is used for acquiring change operation data in response to the triggering of an upstream system change event;
and the adaptation layer change processing module is used for carrying out change processing on the adaptation layer in the data processing system according to the change operation data.
Optionally, when the upstream system change event is an original upstream system modification event, the change operation data includes adaptation layer modification information, and the adaptation layer change processing module includes:
and the adaptation layer modification unit is used for modifying the adaptation layer corresponding to the target original upstream system indicated by the original upstream system modification event according to the adaptation layer modification information.
Optionally, when the upstream system change event is a newly added upstream system event, the change operation data includes adaptation layer configuration information, and the adaptation layer change processing module includes:
and the adaptation layer construction unit is used for constructing an adaptation layer corresponding to the target new upstream system indicated by the newly added upstream system event according to the adaptation layer configuration information.
Optionally, the change data obtaining module includes:
the configuration template providing unit is used for providing an adaptation layer configuration template for a user in response to the newly-added upstream system event being triggered;
the configuration content receiving unit is used for receiving configuration content input by a user based on the adaptation layer configuration template;
and the configuration information determining unit is used for determining the configuration information of the adaptation layer according to the configuration template and the configuration content of the adaptation layer.
Optionally, the adaptation layer is implemented by a structured query language SQL code.
Optionally, the adaptation layer includes a first adaptation logic and a second adaptation logic, where the first adaptation logic includes a mapping rule at a field level between a source table in the source data and a standard interface table in the standard interface data, and the second adaptation logic includes an association rule of at least two source tables in the source data.
Optionally, the adaptation layer constructing unit further includes:
the database type determining subunit is used for determining a target database type corresponding to the data processing system;
and the newly added adaptation layer construction subunit is used for generating an SQL code matched with the type of the target database according to the adaptation layer configuration information so as to construct an adaptation layer corresponding to a target new upstream system indicated by the newly added upstream system event.
Optionally, the apparatus further comprises:
the system comprises a downstream data processing task, a distributed data processing module and a data processing module, wherein the downstream data processing task is triggered by a distributed data processing module;
the downstream system determining module is used for determining at least one downstream system corresponding to the downstream data processing task;
the adaptive data processing module is used for processing the interface data to be issued into adaptive data to be issued based on an adaptive layer corresponding to the at least one downstream system in the data processing system, wherein the adaptive data to be issued meets the business logic requirement corresponding to the at least one downstream system;
and the adaptation data sending module is used for sending the adaptation data to be issued to a corresponding downstream system.
The data processing device provided by the embodiment of the invention can execute the data processing method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
The embodiment of the invention also provides a data processing system which can be used as an upstream system and a downstream system. The data processing system may be adapted to perform the data processing method provided by any of the embodiments of the invention.
Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 5, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 11 can perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from a storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data necessary for the operation of the electronic apparatus 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
A number of components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, or the like; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. The processor 11 performs the various methods and processes described above, such as a face authentication method.
In some embodiments, the face authentication method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the face authentication method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the face authentication method by any other suitable means (e.g. by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on a machine, as a stand-alone software package partly on a machine and partly on a remote machine or entirely on a remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user may provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.
Embodiments of the present invention further provide a computer program product, including a computer program, where the computer program, when executed by a processor, implements the data processing method provided in any embodiment of the present application.
Computer program product in implementing the computer program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C + +, and including conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The data processing device, the equipment, the system, the storage medium and the product provided in the above embodiments can execute the cache updating method provided in any embodiment of the present application, and have corresponding functional modules and beneficial effects for executing the method. For technical details that are not described in detail in the above embodiments, reference may be made to a data processing method provided in any embodiment of the present application.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will appreciate that the present invention is not limited to the particular embodiments described herein, and that various obvious changes, rearrangements and substitutions will now be apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (15)
1. A data processing method is applied to a data processing system and comprises the following steps:
in response to an upstream data processing task being triggered, determining at least one upstream system corresponding to the upstream data processing task;
processing source data in the at least one upstream system into standard interface data based on an adaptation layer corresponding to the at least one upstream system in the data processing system, wherein the standard interface data meet business logic requirements corresponding to the upstream data processing tasks in the data processing system;
and processing the standard interface data based on a preset standard interface processing strategy corresponding to the upstream data processing task to obtain task result data corresponding to the upstream data processing task.
2. The method of claim 1, wherein in the case that the upstream data processing task corresponds to multiple upstream systems, there are at least two upstream systems corresponding to different adaptation layers.
3. The method of claim 1, further comprising:
acquiring change operation data in response to an upstream system change event being triggered;
and changing the adaptation layer in the data processing system according to the change operation data.
4. The method of claim 3, wherein the change operation data comprises adaptation layer modification information if the upstream system change event is an original upstream system modification event;
wherein, the changing the adaptation layer in the data processing system according to the change operation data includes:
and modifying the adaptation layer corresponding to the target original upstream system indicated by the original upstream system modification event according to the adaptation layer modification information.
5. The method of claim 3, wherein in the event that the upstream system change event is a new upstream system event, the change operation data comprises adaptation layer configuration information;
wherein, the changing the adaptation layer in the data processing system according to the change operation data includes:
and constructing an adaptation layer corresponding to the target new upstream system indicated by the newly added upstream system event according to the adaptation layer configuration information.
6. The method of claim 5, wherein obtaining adaptation layer configuration information in response to a new upstream system event being triggered comprises:
in response to a new upstream system event being triggered, providing an adaptation layer configuration template to a user;
receiving configuration content input by a user based on the adaptation layer configuration template;
and determining adaptation layer configuration information according to the adaptation layer configuration template and the configuration content.
7. The method of claim 6, wherein the adaptation layer is implemented by Structured Query Language (SQL) code.
8. The method according to claim 7, wherein the adaptation layer comprises a first adaptation logic and/or a second adaptation logic, wherein the first adaptation logic comprises field-level mapping rules between source tables in the source data and standard interface tables of the standard interface data, and wherein the second adaptation logic comprises association rules of at least two source tables in the source data.
9. The method according to claim 7, wherein the constructing an adaptation layer corresponding to a target new upstream system indicated by the newly added upstream system event according to the adaptation layer configuration information includes:
determining a target database type corresponding to the data processing system;
and generating an SQL (structured query language) code matched with the type of the target database according to the configuration information of the adaptation layer so as to construct the adaptation layer corresponding to the target new upstream system indicated by the newly added upstream system event.
10. The method of claim 1, further comprising:
responding to the triggering of a downstream data processing task, and processing data to be issued in the data processing system based on a preset unified interface processing strategy corresponding to the downstream data processing task to obtain interface data to be issued;
determining at least one downstream system corresponding to the downstream data processing task;
processing the interface data to be issued into adaptation data to be issued based on an adaptation layer corresponding to the at least one downstream system in the data processing system, wherein the adaptation data to be issued meets the business logic requirement corresponding to the at least one downstream system;
and sending the adaptation data to be issued to a corresponding downstream system.
11. A data processing apparatus, characterized by comprising:
the system comprises an upstream system determining module, a data processing module and a data processing module, wherein the upstream system determining module is used for determining at least one upstream system corresponding to an upstream data processing task in response to the upstream data processing task being triggered;
a source data processing module, configured to process, based on an adaptation layer corresponding to the at least one upstream system in the data processing system, source data in the at least one upstream system into standard interface data, where the standard interface data meets a business logic requirement corresponding to the upstream data processing task in the data processing system;
and the interface data processing module is used for processing the standard interface data based on a preset standard interface processing strategy corresponding to the upstream data processing task so as to obtain task result data corresponding to the upstream data processing task.
12. A data processing system for performing the data processing method of any one of claims 1 to 10.
13. An electronic device configured as a data processing system according to claim 12, the electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the data processing method according to any one of claims 1-10 when executing the computer program.
14. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the data processing method of any one of claims 1 to 10.
15. A computer program product comprising a computer program, characterized in that the computer program realizes the data processing method according to any one of claims 1-10 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211185399.6A CN115640345A (en) | 2022-09-27 | 2022-09-27 | Data processing method, device, system, equipment, storage medium and product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211185399.6A CN115640345A (en) | 2022-09-27 | 2022-09-27 | Data processing method, device, system, equipment, storage medium and product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115640345A true CN115640345A (en) | 2023-01-24 |
Family
ID=84942762
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211185399.6A Pending CN115640345A (en) | 2022-09-27 | 2022-09-27 | Data processing method, device, system, equipment, storage medium and product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115640345A (en) |
-
2022
- 2022-09-27 CN CN202211185399.6A patent/CN115640345A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113342345A (en) | Operator fusion method and device of deep learning framework | |
| CN114816393B (en) | Information generation method, device, equipment and storage medium | |
| CN114297119B (en) | Intelligent contract execution method, device, equipment and storage medium | |
| CN115421922A (en) | Current limiting method, device, equipment, medium and product of distributed system | |
| CN116611411A (en) | Business system report generation method, device, equipment and storage medium | |
| CN115344753A (en) | Federal diagram calculation method, federal diagram calculation configuration method, device and electronic equipment | |
| CN114327802B (en) | Method, apparatus, device and medium for block chain access to data outside chain | |
| CN115640345A (en) | Data processing method, device, system, equipment, storage medium and product | |
| CN115456634A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN113110920B (en) | Operation method, device, equipment and storage medium of block chain system | |
| CN114218313A (en) | Data management method, device, electronic equipment, storage medium and product | |
| CN116756460B (en) | Combined data acquisition method and device and related equipment | |
| CN113239296B (en) | Method, device, equipment and medium for displaying small program | |
| CN114780022B (en) | Method and device for realizing additional writing operation, electronic equipment and storage medium | |
| CN113254114B (en) | Operation method, device, equipment and storage medium of block chain system | |
| CN115525614A (en) | Data access method, device, equipment, system and storage medium | |
| CN114860312A (en) | Mobile terminal adaptation method, device, equipment and medium based on credit business | |
| CN116109387A (en) | Electronic contract processing method and device, electronic equipment and medium | |
| CN118012897A (en) | Heterogeneous database grammar conversion method, device, equipment and storage medium | |
| CN116382640A (en) | Method, device, equipment and storage medium for managing micro-service | |
| CN117331944A (en) | Database table generation method, device, equipment and storage medium | |
| CN117632545A (en) | Data sharing method and device of business system and electronic equipment | |
| CN117931506A (en) | Document verification method, device, equipment and storage medium | |
| CN117033473A (en) | Paste source adaptation method, device, equipment, system and storage medium | |
| CN115587091A (en) | Data storage method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |