CN112148711B

CN112148711B - Batch processing task processing method and device

Info

Publication number: CN112148711B
Application number: CN202010997973.2A
Authority: CN
Inventors: 陈伟华
Original assignee: CCB Finetech Co Ltd
Current assignee: CCB Finetech Co Ltd
Priority date: 2020-09-21
Filing date: 2020-09-21
Publication date: 2023-04-25
Anticipated expiration: 2040-09-21
Also published as: CN112148711A

Abstract

The invention discloses a method and a device for processing batch processing tasks, and relates to the technical field of computers. One specific implementation mode of the method comprises the steps that when a preset trigger condition of a batch processing task is met, configuration information of the batch processing task is obtained; and inquiring a database configuration table according to the configuration information, obtaining the service class to be processed by the batch processing task, and further calling a corresponding general service class operation to process the batch processing task. Therefore, the implementation mode of the invention can solve the problems of low development efficiency and complex online of the existing batch processing engineering.

Description

Batch processing task processing method and device

Technical Field

The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for processing a batch processing task.

Background

For each application system, such as credit flow management system, loan system, deposit system, payment settlement system, etc., there are requirements of receiving data of external system, processing data, providing processing data to the service of the system, and providing data of external system at background timing, and these functional requirements are integrated together, namely batch processing engineering. In practical situations, an application system adds a new function, which may depend on data of other systems, and other systems may also need service data of the new function of the system, and the system may also have a requirement of processing data at regular time.

In the process of implementing the present invention, the inventor finds that at least the following problems exist in the prior art:

general batch development may require more coding work and often requires restarting batch engineering when online. Restarting a batch process must take into account whether it will affect the batch task that is currently running, and whether it will affect the batch task that is about to run. Thus, each time an online occurs, a lot of analysis work or relatively complex online deployment operations are required.

Disclosure of Invention

In view of the above, the embodiment of the invention provides a method and a device for processing batch processing tasks, which can solve the problems of low development efficiency and complex online of the existing batch processing engineering.

In order to achieve the above object, according to an aspect of the embodiments of the present invention, there is provided a method for processing a batch task, including acquiring configuration information of the batch task when a preset trigger condition of the batch task is monitored to be satisfied; and inquiring a database configuration table according to the configuration information, obtaining the service class to be processed by the batch processing task, and further calling a corresponding general service class operation to process the batch processing task.

Optionally, the method comprises:

and monitoring the starting time of the batch processing task to reach or the data processing of an upstream system on which the batch processing task depends to finish, and acquiring the configuration information of the batch processing task.

Optionally, invoking the corresponding generic service class job to process the batch task, including:

if the called general service class operation is a general connection operation, reading and analyzing the DDL file according to the full path of the DDL file configured by the database, generating a configuration file loaded by the database file, and further calling the service loading file of the database based on the configuration file to be put into a ground table generated according to the DDL file.

Optionally, the service loading file of the configuration file calling database is put into a ground table generated according to the DDL file, including:

judging whether a first target field in a database configuration table is empty, if not, calling a service loading file of the database to be put into a ground table generated according to a DDL file, and executing an SQL statement configured by the first target field so as to process the ground table into the target table; and if so, processing the land table to a target table by using a corresponding processing model according to the type of the second target field in the database configuration.

Optionally, processing the land table to the target table using a corresponding processing model according to a type of the second target field in the database configuration, including:

judging whether the type of a second target field in the database configuration is full data, if so, adopting a trunk statement to process a target table, and then inserting the data in the falling table into the target table through an insert statement; if not, updating the increment data in the falling table to the target table by using a merge statement, or inserting the increment data in the falling table into the target table by using an insert statement.

if the called general service class operation is a general supply number operation, reading preset supply number configuration table information;

and unloading corresponding data from the database table according to the supply number configuration table information and transmitting the data to a downstream system.

Optionally, before reading the preset supply configuration table information, the method includes:

according to the configuration information of the batch processing task, obtaining a to-be-supplied table identifier, a supply number time point, a supply number data range and a supply number standard path, and generating a configuration table of a supply table level;

and obtaining the field names and the field types of the number to be supplied, and generating a configuration table of the field level of the supply table.

In addition, the invention also provides a batch processing task processing device, which comprises a scheduling module, a batch processing engineering service module and a processing module, wherein the scheduling module is used for monitoring the preset triggering condition of the batch processing task and sending a request carrying configuration information of the batch processing task to the batch processing engineering service module;

and the batch engineering service module is used for receiving the request, inquiring the database configuration table based on the configuration information of the batch task in the request, acquiring the service class to be processed by the batch task, and further calling the corresponding general service class operation to process the batch task.

One embodiment of the above invention has the following advantages or benefits: aiming at batch processing demand tasks, such as receiving newly added external system data, adding newly added supply numbers of the system table and not very complex batch processing demands of logic, developers can quickly develop the batch processing demand tasks without restarting batch processing services (the batch processing does not involve user operation, and the system executes data processing logic according to a specified time or a specified dependent method), various problems occurring during online are reduced, and the system stability is enhanced. Therefore, the configurable batch processing operation online without restarting is realized, and the universal logic of the configurable connection (the system receives the data file transmitted from the upstream, stores and processes the data file, and is called connection for short) and the universal logic of the configurable supply (the data of the system is provided for the downstream system in the form of file or other forms, and is called supply for short) is realized.

Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.

Drawings

The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:

FIG. 1 is a schematic diagram of the main flow of a method for processing a batch task according to a first embodiment of the present invention;

FIG. 2 is a schematic diagram of upstream system data transmission according to an embodiment of the present invention;

FIG. 3 is a workflow diagram of a generic join job according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of data transmission to a downstream system according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of the main flow of a method for processing a batch task according to a second embodiment of the present invention;

FIG. 6 is a schematic diagram of the main modules of a processing apparatus for batch processing tasks according to an embodiment of the invention;

FIG. 7 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

fig. 8 is a schematic diagram of a computer system suitable for use in implementing an embodiment of the invention.

Detailed Description

Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Fig. 1 is a schematic diagram of main flow of a processing method of a batch task according to a first embodiment of the present invention, and as shown in fig. 1, the processing method of a batch task includes:

step S101, acquiring configuration information of a batch processing task when the preset trigger condition of the batch processing task is met.

In some embodiments, the completion of the start time of the batch task or the data processing of an upstream system on which the batch task depends is monitored, and configuration information of the batch task is obtained.

Step S102, inquiring a database configuration table according to the configuration information, obtaining the service class to be processed of the batch processing task, and further calling a corresponding general service class operation to process the batch processing task.

In some embodiments, if the generic service class job called when executing step S103 is a generic join job, the DDL file is read and parsed according to the full path of the DDL (database schema definition language) file configured by the database to generate a configuration file loaded by the database file (e.g., the oracle database generates a control file of sqlldr). And then, the service loading file of the database is called based on the configuration file and is put into a ground table generated according to the DDL file.

As shown in fig. 2, the data of the upstream system data transfer instant connection process includes two types of files: a data file and a field definition file. The data file is divided according to a certain symbol, and each divided data segment corresponds to each field at one time. And the other is divided by length, i.e. how many bits to how many bits represent a certain field. The data definition file is a definition of the fields, that is, defines which fields are shared, the format of each field, the location to which the field belongs, and so on. Therefore, the invention presets the database configuration according to the data transmitted in the connection process, and specifically comprises the following main fields:

FILE_TYPE	file type
		LOAD_TYPE	Load type
TABLE_NAME	Target table name
		DAT_FILE_PATH	DAT file full path
DDL_FILE_PATH	DDL file full path
		FIELD_DCLIMITER	Partition character
UPDATE_COLUMN	Specially updated field names
		UPDATE_VALUE	Specially updated field values
INSERT_COLUMN	Specially inserted field names
		INSERT_VALUE	Specially inserted field values
FALL_TABLE_FILTER	Temporary table filtering conditions
		SQL_BEFORE_PROCESS_TARGET	Pre-executed SQL before processing to target table

It should be noted that the logic of the general service class operation is a common class code, and no modification is needed after the first online. If the number of the connection of a new table is needed, a developer does not need to develop codes, only needs to configure a record in the database, and the program can finish the number connection according to the configuration information of the database and the data file and the field definition file provided at the upstream.

Further embodiments, as shown in FIG. 3, determine whether a first TARGET field (e.g., SQL_BEFORE_PROCESS_TARGET field) in a database configuration table is empty, and if not, call a service loading file of the database to be put into a drop table generated according to the DDL file, execute an SQL statement configured by the first TARGET field to PROCESS the drop table into the TARGET table. If so, the floor table is processed to the target table by using the corresponding processing model according to the TYPE of the second target field (for example, the load_type field) in the database configuration.

In a further embodiment, processing the floor table to the target table using the corresponding processing model according to the type of the second target field in the database configuration includes:

and judging whether the type of the second target field in the database configuration is full data, wherein the type of the second target field is the data type transmitted by an upstream system and comprises the full data and incremental data.

If yes, processing the target table by adopting a trunk statement, and then inserting the data in the falling table into the target table by an insert statement. That is, if the upstream system transmits the full amount of data, ALL is configured in the database configuration, the target table is processed by adopting the trunk statement, and then the data on the ground is inserted into the target table. In the process of inserting the data of the ground table into the target table, the data with the same field name type as the target table is inserted into the target table. And processing the data according to processing rules preset in the database configuration and then inserting the processed data into the target table. For example, some specially inserted fields may be configured according to the INSERT_COLUMN and INSERT_VALUE fields in the database configuration, for example, VALUEs that require simple processing of constants, sequences, and original fields may be inserted into certain fields of the target table.

If not, updating the increment data in the falling table to the target table by using a merge statement, or inserting the increment data in the falling table into the target table by using an insert statement. That is, if the data transmitted by the upstream system is incremental data, different processing methods may be adopted according to different storage modes: one way to save is to update the data of the upstream system change into the target table (corresponding to the add_merge field in the database configuration), then the MERGE statement is used to update the drop table into the target table. Wherein, the data with the same field name type as the target table is directly updated to the target table. And processing the data according to processing rules preset in the database configuration, and updating the data to the target table. Some special fields may be configured, for example, in the database configured INSERT_ COLUMN, INSERT _VALUE, UPDATE_ COLUMN, UPDATE _VALUE fields.

One is to INSERT the data of the current day change of the upstream system directly into the target table (corresponding to the add_insert field in the database configuration), and to INSERT the data of the destination table directly into the target table. The data with the same field name type as the target table is automatically inserted into the target table, the data with the different field name type as the target table is automatically inserted into the target table, and some special fields for inserting UPDATE can be configured in the INSERT_ COLUMN, INSERT _VALUE and UPDATE_ COLUMN, UPDATE _VALUE fields in the database configuration table, and then the data is processed for insertion into the target table.

In other embodiments, if the universal service class job called in the step S103 is a universal provisioning job, the preset provisioning configuration table information is read, and then the corresponding data is unloaded from the database table and transmitted to the downstream system according to the provisioning configuration table information.

Further, before the preset provisioning configuration table information is read, a pre-generated provisioning configuration table needs to be configured, and as shown in fig. 4, two files are provided for data transmission in the provisioning process, one is a data file, and the other is a field definition file. The data file generally has two formats, one is divided according to a certain symbol, and each segment of divided data corresponds to each field at one time; one is to divide by length, i.e. how many bits to how many bits represent a certain field. And the data definition file refers to the definition of the fields, that is, defines which fields are shared, the format of each field, the location to which the field belongs, and so on.

Therefore, according to the data transmission condition in the provisioning process, a configuration table of the provisioning table level and a configuration table of the provisioning table field level are configured, namely the provisioning configuration table comprises the configuration table of the provisioning table level and the configuration table of the provisioning table field level, and specifically comprises the following steps:

and acquiring a to-be-supplied table identifier, a supply number time point, a supply number data range and a supply number standard path according to the configuration information of the batch processing task, and generating a configuration table of a supply table level. And obtaining the field names and the field types of the number to be supplied, and generating a configuration table of the field level of the supply table.

That is, the developer only needs to configure the configuration table of the table level and the configuration table of the table field level, and the program will automatically read the configuration table of the table level and the configuration table of the table field level, automatically unload the data of the database table, and then transmit the data to the downstream system.

Fig. 5 is a schematic diagram of the main flow of a processing method of a batch task according to a second embodiment of the present invention, where information such as a job name and a called service of a batch task needs to be configured in a database. The HTTP request can be sent by carrying information such as job name and time after monitoring that the starting time of the batch processing task reaches or the data processing of the upstream system on which the batch processing task depends is completed (i.e. the batch processing task can be executed after the execution of other batch processing tasks is completed). And receiving the HTTP request, analyzing the carried HTTP request parameters to obtain information such as job names, and inquiring configuration information in the database configuration to obtain service classes to be processed. And then, according to the service class configured by the database, calling corresponding universal supply number operation, universal receiving operation, universal storage procedure calling service or universal script calling service which do not need to be developed through a batch engineering unified interface (i.e. an entry).

It can be seen that the batch engineering has a set of common batch code, the batch development is not needed in JAVA, the batch processing logic can be developed on the storage process, shell script, perl script, etc., and the deployment of the storage engineering and various scripts is not needed to restart JAVA service. Therefore, the batch development method which only needs to be configured and does not need to restart the batch service is realized.

FIG. 6 is a schematic diagram of main modules of an apparatus for processing a batch task according to an embodiment of the present invention, and as shown in FIG. 6, the apparatus 600 for processing a batch task includes a scheduling module 601 and a batch engineering service module 602. The scheduling module 601 monitors that a preset triggering condition of a batch processing task is met, and sends a request carrying configuration information of the batch processing task to a batch processing engineering service module; the batch engineering service module 602 receives the request, queries a database configuration table based on configuration information of a batch task in the request, obtains a service class to be processed by the batch task, and further invokes a corresponding general service class job to process the batch task.

In some embodiments, the scheduling module 601 is configured to:

In some embodiments, the batch engineering service module 602 invokes a corresponding generic service class job to process a batch task, including:

In some embodiments, the batch engineering service module 602 imports a service load file of the database into a drop table generated from DDL files based on the configuration file call database, including:

In some embodiments, the batch engineering service module 602 processes the floor table to a target table using a corresponding processing model according to the type of the second target field in the database configuration, including:

In some embodiments, before the batch engineering service module 602 reads the preset supply configuration table information, the method includes:

In the method for processing a batch task and the apparatus for processing a batch task according to the present invention, the specific implementation content has a corresponding relationship, and therefore, the repetitive content will not be described.

FIG. 7 illustrates an exemplary system architecture 700 of a batch task processing method or a batch task processing device to which embodiments of the present invention may be applied.

As shown in fig. 7, a system architecture 700 may include

terminal devices

701, 702, 703, a network 704, and a server 705. The network 704 is the medium used to provide communication links between the

terminal devices

701, 702, 703 and the server 705. The network 704 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

A user may interact with the server 705 via the network 704 using the

terminal devices

701, 702, 703 to receive or send messages or the like. Various communication client applications such as shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only) may be installed on the

terminal devices

701, 702, 703.

The

terminal devices

701, 702, 703 may be various electronic devices having a processing screen for batch processing tasks and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.

The server 705 may be a server providing various services, such as a background management server (by way of example only) providing support for shopping-type websites browsed by users using the

terminal devices

701, 702, 703. The background management server may analyze and process the received data such as the product information query request, and feedback the processing result (e.g., the target push information, the product information—only an example) to the terminal device.

It should be noted that, the method for processing batch processing tasks provided in the embodiment of the present invention is generally executed by the server 705, and accordingly, the computing device is generally disposed in the server 705.

It should be understood that the number of terminal devices, networks and servers in fig. 7 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Referring now to FIG. 8, there is illustrated a schematic diagram of a computer system 800 suitable for use in implementing an embodiment of the present invention. The terminal device shown in fig. 8 is only an example, and should not impose any limitation on the functions and the scope of use of the embodiment of the present invention.

As shown in fig. 8, the computer system 800 includes a Central Processing Unit (CPU) 801 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 802 or a program loaded from a storage section 808 into a Random Access Memory (RAM) 803. In the RAM803, various programs and data required for the operation of the computer system 800 are also stored. The CPU801, ROM802, and RAM803 are connected to each other by a bus 804. An input/output (I/O) interface 805 is also connected to the bus 804.

The following components are connected to the I/O interface 805: an input portion 806 including a keyboard, mouse, etc.; an output section 807 including a processor (LCD) or the like such as a Cathode Ray Tube (CRT), a liquid crystal batch processing task, and a speaker or the like; a storage section 808 including a hard disk or the like; and a communication section 809 including a network interface card such as a LAN card, a modem, or the like. The communication section 809 performs communication processing via a network such as the internet. The drive 810 is also connected to the I/O interface 805 as needed. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 810 as needed so that a computer program read out therefrom is mounted into the storage section 808 as needed.

In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication section 809, and/or installed from the removable media 811. The above-described functions defined in the system of the present invention are performed when the computer program is executed by a Central Processing Unit (CPU) 801.

The computer readable medium shown in the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules involved in the embodiments of the present invention may be implemented in software or in hardware. The described modules may also be provided in a processor, for example, as: a processor includes a dispatch module and a batch processing engineering service module. The names of these modules do not constitute a limitation on the module itself in some cases.

As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs, and when the one or more programs are executed by the equipment, the equipment is caused to comprise the steps of acquiring configuration information of a batch task when the preset trigger condition of the batch task is monitored; and inquiring a database configuration table according to the configuration information, obtaining the service class to be processed by the batch processing task, and further calling a corresponding general service class operation to process the batch processing task.

According to the technical scheme provided by the embodiment of the invention, the problems of low development efficiency and complex online of the existing batch engineering can be solved.

The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.

Claims

1. A method for processing a batch processing task, comprising:

acquiring configuration information of a batch processing task when the preset trigger condition of the batch processing task is monitored to be met;

inquiring a database configuration table according to the configuration information, obtaining a service class to be processed by the batch processing task, and further calling a corresponding general service class operation to process the batch processing task;

the method for calling the corresponding general service class job to process the batch processing task comprises the following steps: if the called general service class operation is a general connection operation, reading and analyzing the DDL file according to the whole path of the DDL file configured by the database, generating a configuration file loaded by the database file, and calling the service loading file of the database based on the configuration file to be put into a ground table generated according to the DDL file;

and the service loading file of the database is called based on the configuration file to be put into a ground table generated according to the DDL file, comprising the following steps: judging whether a first target field in a database configuration table is empty, if not, calling a service loading file of the database to be put into a ground table generated according to a DDL file, and executing an SQL statement configured by the first target field so as to process the ground table into the target table; if yes, processing the land table to a target table by using a corresponding processing model according to the type of the second target field in the database configuration, including: and calling a corresponding statement to insert the data or the incremental data in the falling table into the target table by judging whether the type of the second target field in the database configuration is full data.

2. The method according to claim 1, characterized in that it comprises:

3. The method of claim 1, wherein machining the floor table to the target table using the corresponding machining model according to the type of the second target field in the database configuration comprises:

4. The method of claim 1, wherein invoking the corresponding generic service class job to process the batch task comprises:

5. The method of claim 4, wherein before reading the preset number of supplies configuration table information, comprising:

6. A batch processing task processing apparatus, comprising:

the scheduling module is used for monitoring that preset triggering conditions of the batch processing tasks are met and sending a request carrying configuration information of the batch processing tasks to the batch processing engineering service module;

the batch engineering service module is used for receiving a request, inquiring a database configuration table based on configuration information of batch tasks in the request, acquiring service classes to be processed of the batch tasks, and further calling corresponding general service class jobs to process the batch tasks;

the batch engineering service module calls corresponding general service class operation to process batch tasks, and the method comprises the following steps: if the called general service class operation is a general connection operation, reading and analyzing the DDL file according to the whole path of the DDL file configured by the database, generating a configuration file loaded by the database file, and calling the service loading file of the database based on the configuration file to be put into a ground table generated according to the DDL file;

7. An electronic device, comprising:

one or more processors;

storage means for storing one or more programs,

when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-5.

8. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-5.