CN110895490A - Data batch processing system, method, equipment and storage medium - Google Patents
Data batch processing system, method, equipment and storage medium Download PDFInfo
- Publication number
- CN110895490A CN110895490A CN201911198742.9A CN201911198742A CN110895490A CN 110895490 A CN110895490 A CN 110895490A CN 201911198742 A CN201911198742 A CN 201911198742A CN 110895490 A CN110895490 A CN 110895490A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- batch processing
- task
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 285
- 238000000034 method Methods 0.000 title claims description 32
- 230000002159 abnormal effect Effects 0.000 claims description 41
- 238000012544 monitoring process Methods 0.000 claims description 17
- 238000011282 treatment Methods 0.000 claims description 4
- 238000004891 communication Methods 0.000 claims description 2
- 230000000977 initiatory effect Effects 0.000 claims 1
- 238000003672 processing method Methods 0.000 abstract description 12
- 238000007726 management method Methods 0.000 description 26
- 230000005856 abnormality Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000006243 chemical reaction Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000004590 computer program Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 230000000246 remedial effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a data batch processing system, a data batch processing method, data batch processing equipment and a storage medium. The system comprises: the data reading module is used for converting the original data in the batch processing task into a first preset data format and reading the original data in the first preset data format according to a reading command in the batch processing task; the data processing module is used for processing the original data based on the processing command in the batch processing task and storing the processed data into a preset list; and the data writing module is used for converting the data in the preset list into a second preset data format according to the writing command in the batch processing task and writing the data in the second preset data format into the target storage container. The data reading module, the data processing module and the data writing module are packaged independently. The high maintainability and reusability of the code are realized in a configuration mode, and the batch processing of the data is realized in a simple, high-efficiency and low-cost mode.
Description
Technical Field
The embodiment of the invention relates to an information data processing technology, in particular to a data batch processing system, a data batch processing method, data batch processing equipment and a storage medium.
Background
With the increasing explosion and development of large data, the data volume is more and more, and when a large amount of data is processed, batch processing is particularly important, and the batch processing refers to a processing mode in which a computer program executes a series of tasks based on the input of a batch without manual intervention.
At present, a batch processing system is generally only used for processing simple scenes, namely, at the initial stage of online project, due to the fact that batch tasks are few, the work is simple, the batch processing system can only basically meet the requirements, but after the online project is completed, the batch tasks are more and more, the scenes are more and more complex, the original batch processing system is more and more careless, and finally, only the redesign is carried out by pushing over, time and labor are wasted, and the requirements of batch processing cannot be met.
Disclosure of Invention
Embodiments of the present invention provide a data batch processing system, method, device, and storage medium, which implement high maintainability and reusability of codes in a configuration manner, and implement batch processing of data in a simple, efficient, and low-cost manner.
In a first aspect, an embodiment of the present invention provides a data batch processing system, where the system includes:
the device comprises a data reading module, a data processing module and a data writing module, wherein the data reading module, the data processing module and the data writing module are respectively and independently packaged;
the data reading module is used for converting the original data in the batch processing task into a first preset data format and reading the original data in the first preset data format according to a reading command in the batch processing task;
the data processing module is used for processing the original data based on the processing command in the batch processing task and storing the processed data into a preset list;
and the data writing module is used for converting the data in the preset list into a second preset data format according to the writing command in the batch processing task and writing the data in the second preset data format into the target storage container.
In a second aspect, an embodiment of the present invention further provides a data batch processing method, where the method includes:
converting original data into a preset data format according to a read command in the batch processing task, and reading the original data in the preset data format;
processing the original data based on the processing command in the batch processing task, and storing the processed data into a preset list;
and converting the data in the preset list into a second preset data format according to the write command in the batch processing task, and writing the data in the second preset data format into the target storage container.
In a third aspect, an embodiment of the present invention further provides an apparatus, where the apparatus includes:
one or more processors;
storage means for storing one or more programs;
when the one or more programs are executed by the one or more processors, the one or more processors implement a data batch processing method according to any one of the embodiments of the present invention.
In a fourth aspect, the present invention further provides a storage medium containing computer-executable instructions, where the computer-executable instructions, when executed by a computer processor, are used to implement a data batch processing method according to any one of the embodiments of the present invention.
The embodiment of the invention respectively and independently encapsulates the data reading module, the data processing module and the data writing module, and when the data is processed in batch, the reading, the processing and the writing of the original data in the batch processing task are respectively and independently executed, but the codes for reading, the processing and the writing of the data are not written together, so when one or more of the codes for reading, the processing and the writing of the original data are required to be modified, only the codes required to be modified are required to be modified, and all the codes are not required to be rewritten, and the high maintainability and the reusability of the codes are realized in a configuration mode; the data reading module can convert the original data in the batch processing task into a data format which can be read by a system and read the data; the data processing module is used for processing the read original data based on a processing command in the batch processing task and storing the processed data into a preset list; the data writing module is used for converting data in the preset list into a data format which can be stored in the target storage container according to a writing command in the batch processing task, and writing the data into the target storage container, and the data processing module and the data writing module can convert the data format, so that the data can be conveniently read and written, the problem that the data can not be read and written because the data format does not meet the reading and writing requirements is solved, and the compatibility of a plurality of data formats is improved.
Drawings
FIG. 1 is a schematic structural diagram of a data batch processing system according to a first embodiment of the present invention;
FIG. 2 is a flowchart illustrating the execution of batch processing tasks according to one embodiment of the present invention;
FIG. 3 is a flowchart of a data batch processing method according to a second embodiment of the present invention;
fig. 4 is a schematic structural diagram of an apparatus in a third embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a schematic structural diagram of a data batch processing system according to an embodiment of the present invention, as shown in fig. 1, the system includes: the data reading module 11, the data processing module 12 and the data writing module 13, wherein the data reading module 11, the data processing module 12 and the data writing module 13 are packaged independently.
For example, when data is processed in batches, the original data in the received batch task needs to be read from the original storage container, the read data is processed, and the processed data is written into the target container, and for example, the data batch processing operation may include, but is not limited to, the following processing modes of the data: the data reading module 11, the data processing module 12 and the data writing module 13 may respectively and independently read, process and write the original data in the batch task.
Optionally, the data reading module 11 is configured to convert the raw data in the batch processing task into a first preset data format, and read the raw data in the first preset data format according to a read command in the batch processing task.
Illustratively, the batch tasks are batch tasks to be processed, the batch tasks may be one or more, and the batch tasks may be provided by a user or extracted from an external database. The raw data is data in a batch processing task, for example, the raw data may be bank transfer flow, and the like, the raw data may be stored in a raw storage container in a form of a file, and when the raw data stored in the raw storage container is read, and when the data format of the raw data stored in the raw storage container cannot be read by a system, the data reading module 11 may convert the data format of the raw data into a data format that can be read by the system, that is, a first preset data format, and read the raw data in the first preset data format according to a read command (a command to read the raw data) in the batch processing task. For example, the original data is stored in the original storage container in a file format, where the file format may be a CSV format, but the system cannot read the CSV file format, the data reading module 11 converts the CSV file format containing the original data into a JSON file format that can be read, and reads the original data converted into the JSON file format according to the read command. Optionally, when the initial data format of the original data is the first preset data format, it is not necessary to perform data format conversion on the original data.
Optionally, the data processing module 12 is configured to process the original data based on the processing command in the batch processing task, and store the processed data in a preset list.
For example, the preset list is a preset storage list in the system, and is used for storing the processed data, and the preset list may be a file or a memory. The processing command carries a processing mode of the original data converted into the first preset data format, for example, the data may be converted into a case form, or the data may be calculated according to a certain rule. And processing the original data in the first preset data format according to the processing command in the batch processing task, and storing the processed data in a preset list. For example, the processing command is to convert the original data in the first preset data format from arabic numbers to capital numbers, where the original data has 123, 2, and 74, and the data processing module 12 converts the data 123, 2, and 74 into one, two, and seven, and stores the processed data in a preset storage file.
Optionally, the data writing module 13 is configured to convert the data in the preset list into a second preset data format according to the write command in the batch processing task, and write the data in the second preset data format into the target storage container.
The target storage container is, for example, a location where data stored in the preset list is written, and may be a memory or a file capable of storing data. When data stored in the preset list is written into the target storage container, and when the data format of the data stored in the preset list does not conform to the data format of the target storage container, the data writing module 13 converts the data in the preset list into the data format conforming to the storage of the target storage container, that is, a second preset data format, according to a write command (a command for writing the data into the target storage container) in the batch processing task, and writes the data in the second preset data format into the target storage container, where the second preset data format is a data format suitable for the target storage container. For example, if the preset list is a file, the file format may be JSON, but the target storage container cannot store the JSON file format, the data writing module 13 converts the JSON file format containing the data in the second preset data format into a CSV file format storable in the target storage container, and writes the data converted into the CSV file format into the target storage container according to the write command. Optionally, when the format of the preset list is the second preset data format, data format conversion does not need to be performed on the preset list.
It should be noted that the system may also preset a default data format that can be read and/or written, when the data format of the read and/or written data is the default data format preset by the system, the system may directly read and/or write the original data, and when the data format of the read and/or written data is not the default data format preset by the system, the data reading module 11 and/or the data writing module 13 converts the data format into a data format that can be read and/or written.
In the technical solution of the above embodiment, the data reading module 11, the data processing module 12, and the data writing module 13 are provided, which have the advantages that data reading, processing, and writing can be separately and independently packaged, and the three do not interfere with each other, when one or more codes need to be modified, the codes that need to be modified can be independently modified, and the codes that do not need to be modified in the codes that need to be read, processed, and written need not be rewritten, and when other programmers need to use one or more modules in the system, the needed modules can be directly called, and the codes in the modules can be adaptively modified, so that high maintainability and reusability of the codes are realized in a configuration manner, the work tasks of the programmers are reduced, the work efficiency is improved, and the cost is reduced. Meanwhile, the data reading module 11 and the data writing module 13 can also convert the data format, so that the data can be conveniently read and written, the problem that the data format cannot be read and/or written because the data format does not meet the reading and/or writing requirements is solved, and the compatibility of a plurality of data formats is improved.
Optionally, on the basis of the foregoing embodiment, the system further includes: and the task management module is used for receiving batch processing tasks and controlling the data reading module 11, the data processing module 12 and the data writing module 13 to execute the batch processing tasks according to the batch processing tasks, wherein the task management module is independently packaged, and the batch processing tasks comprise original data, read commands, processing commands and write commands. Illustratively, referring to fig. 2, fig. 2 is a schematic diagram illustrating an execution flow of a batch processing task according to an embodiment of the present invention. The task management module sequentially sends the read command, the processing command and the write command to the data reading module 11, the data processing module 12 and the data writing module 13 so as to control the data reading module 11, the data processing module 12 and the data writing module 13 to execute the corresponding commands, and thus, the batch processing task is completed.
For example, the task management module may be separately packaged and configured to receive a batch task, and the data reading module 11, the data processing module 12, and the data writing module 13 may be controlled according to the batch task to process the batch task, where the batch task includes raw data to be batch processed, a command to read the raw data, a command to process data, and a command to write data into a target storage container.
Optionally, the task management module is further configured to: and generating a read command according to the initial storage position of the original data in the batch processing task, generating a processing command according to the processing requirement in the batch processing task, and generating a write command according to the target storage position of the data in the batch processing task.
For example, the task management module may generate a read command based on an initial storage location of raw data in the batch task. The task management module may also generate a processing command according to a processing requirement in the batch processing task, where the processing requirement may be a processing mode of the original data, and the processing requirement may be carried in the batch processing task, a requirement provided by a user, or a default requirement of the system. The task management module may also generate a write command based on a target storage location of data in the batch task. For example, the original data in the batch task are: 123. 2 and 74, the initial storage location of the original data is a memory a, the task management module may generate a read command, where the read command is: the original data is read from memory a. The processing requirement in the batch processing task is to convert the arabic numbers of the original data into capital digits, and then the task management module can generate a processing command according to the processing requirement in the batch processing task, where the processing command is: data 123, 2, and 74 are converted to upper case numbers. And if the target storage position of the data in the batch processing task is a memory B, the task management module can generate a write command according to the target storage position of the data in the batch processing task, wherein the write command is as follows: the processed data is written into the memory B.
In the technical solution of the above embodiment, the advantage of setting the task management module is that the data batch processing system can be adapted to batch processing tasks in different scenarios by converting the batch processing tasks into a combination of a read command, a processing command and a write command, without modifying and adjusting the data batch processing system, thereby reducing dependency of the batch processing tasks on the data batch processing system, improving applicability and processing range of the data batch processing system, and reducing maintenance cost of the data batch processing system. Furthermore, the task management module controls the data reading module 11, the data processing module 12 and the data writing module 13 to execute the batch processing task according to the generated reading command, the processing command and the writing command, so that the working efficiency is improved, extra system managers do not need to carry out the previous work of the reading command, the processing command and the writing command on the batch processing task according to the batch processing task, the cost is reduced, meanwhile, the task management module is independently packaged, therefore, when the codes in the task management module need to be modified, only the task management module can be modified without rewriting the codes of other modules in the system, and when other programmers need to use one or more modules in the system, the modules can be directly used, the codes in the modules can be adaptively modified, and the working time of the programmers is saved, the working efficiency is improved, and meanwhile, the high maintainability and reusability of the codes are realized.
Optionally, on the basis of the foregoing embodiment, the system may further include: the data monitoring module is respectively in communication connection with the data processing module and the data writing module and is used for monitoring whether the data has abnormal conditions in the execution process of the writing command; if abnormal conditions occur, alarm information is generated and stored in an information memory, wherein the alarm information comprises: abnormal data and abnormal causes, the abnormal causes including at least one of: the data format exception and the network exception; if no abnormal condition exists, generating processing state information of each batch processing task according to the processing state of each batch processing task; wherein the processing state comprises: the untreated, treated and neutralized treatments were completed.
In the process of writing the data in the second preset data format into the target storage container, for example, an abnormal condition may occur, and data cannot be written into the target storage container, when an abnormal condition occurs, the data monitoring module generates alarm information, and the alarm information includes specific data with abnormality and an abnormal reason, where the abnormal reason may be that data format is not successfully converted in the data format conversion process, data cannot be written into the target storage container due to the abnormal data format, or data cannot be written into the target storage container due to network abnormality in the process of writing data into the target storage container, for example, in the process of writing the data in the second preset data format into the target storage container, the data monitoring module monitors that the data storage is abnormal, and then alarm information is generated, wherein the alarm information is as follows: abnormal data: 74, cause of abnormality: the data format is abnormal. When the batch processing task processing personnel sees the alarm information, the abnormal data can be checked, and a skip command can be set in the abnormal data, so that when data is written, the abnormal data can be skipped, and other data can be directly written. If the abnormal reason is the network abnormality, the batch processing task processing personnel can repeatedly execute the writing operation when seeing the alarm information, and the high robustness of the system is improved.
For example, when no exception occurs during writing the data in the second preset data format into the target storage container, the data monitoring module may generate processing state information of each batch task according to a processing state of each batch task, that is, whether each batch task is unprocessed, in progress, or completed, for example, there are 3 batch tasks, that is, task 1, task 2, and task 3, task 1 is completed, task 2 is being processed, and task 3 is not yet processed, and when task 2 is being processed, the data monitoring module does not monitor data exception before that, the data monitoring module may generate processing state information of the 3 batch tasks, where the processing state information is: task 1, processing is completed; task 2, in processing; task 3, unprocessed.
It should be noted that the alarm information including the abnormal data and the reason of the abnormal may be stored in a separate information storage by the data monitoring module, or may be directly stored in the data monitoring module, and the processing state information of each generated batch processing task may also be stored in a separate information storage, or may be directly stored in the data monitoring module, which is not limited herein.
In the technical scheme of the embodiment, the data monitoring module is arranged, so that whether abnormal conditions occur to the data in the write command execution process can be monitored, if abnormal conditions occur, alarm information can be generated, a processing person of a batch processing task can check the data in real time according to the alarm information or ask a programmer to take remedial measures without influencing subsequent execution operation, and the batch processing efficiency of the system is improved; when no abnormal condition occurs, the data monitoring module can generate the processing state information of each batch processing task, so that the processing personnel of the batch processing tasks can know the processing state of the batch processing tasks in real time conveniently, the processing state information of each batch processing task does not need to be inquired additionally, the working efficiency is improved, and good user experience is provided.
Optionally, on the basis of the foregoing embodiment, the system may further include: the job starting module sends a starting instruction of the batch processing task to the task management module based on the received batch processing task; and the task management module controls the data reading module, the data processing module and the data writing module to execute batch processing tasks according to the starting instruction.
Illustratively, the start instruction is an instruction for the system to execute a received task, according to which the system can start and execute the task, the job starting module sends a start instruction of the batch processing task to the task management module based on the received batch processing task, and according to the start instruction, the task management module can control the data reading module 11, the data processing module 12 and the data writing module 13 to start executing the batch processing task.
Optionally, the job starting module is further configured to restart the batch processing task according to the execution failure node of the batch processing task when the batch processing task fails to be executed.
For example, when the data reading module 11, the data processing module 12 and the data writing module 13 fail to execute the batch task due to an abnormality occurring during the execution of the batch task, the job starting module may restart the execution of the batch task at a place where the batch task failed to execute, where the execution of the batch task was started at the last place where the execution of the batch task failed when the execution of the batch task was restarted, instead of restarting the entire batch task from the beginning of the execution of the batch task, for example, during the data processing, the batch task failed to execute due to a network abnormality, the job starting module may restart the execution of the batch task at a part of the data processing module instead of starting the entire batch task from the beginning at the data reading module.
In the technical solution of the above embodiment, the advantage of providing the job starting module is that, when the task management module receives the starting instruction sent by the job starting module, the task management module controls the data reading module 11, the data processing module 12 and the data writing module 13 to start executing the batch processing task, so that the system can execute orderly, without starting the data reading module 11, the data processing module 12 and the data writing module 13 to start executing the batch processing task when no batch processing task needs to be processed, thereby saving resources, and the job starting module also starts executing the batch processing task when the batch processing task fails to be executed, restarting the batch task at the execution failure node of the batch task, instead of executing the batch task from the beginning, therefore, the system execution efficiency is greatly improved, and the processing efficiency of batch processing tasks is improved.
Optionally, on the basis of the technical solution of the above embodiment, the system may further set a multithread processing mode, and may perform parallel processing on a plurality of batch processing tasks, for example, there are 3 tasks, task 1, task 2, and task 3, and the system may simultaneously read, process, and write the 3 tasks, thereby improving the execution efficiency of the system.
According to the technical scheme, the data reading module, the data processing module and the data writing module are respectively and independently packaged, when data are processed in batches, reading, processing and writing of original data in a batch processing task are respectively and independently executed, and reading, processing and writing codes of the data are not written together, so that when one or more of the codes of the original data, which need to be modified, only the codes which need to be modified are required to be modified, all the codes do not need to be rewritten, and high maintainability and reusability of the codes are realized in a configuration mode; the data reading module can convert the original data in the batch processing task into a data format which can be read by a system and read the data; the data processing module is used for processing the read original data based on a processing command in the batch processing task and storing the processed data into a preset list; the data writing module is used for converting data in the preset list into a data format which can be stored in the target storage container according to a writing command in the batch processing task, and writing the data into the target storage container, and the data processing module and the data writing module can convert the data format, so that the data can be conveniently read and written, the problem that the data can not be read and written because the data format does not meet the reading and writing requirements is solved, and the compatibility of a plurality of data formats is improved.
Example two
Fig. 2 is a flowchart of a data batch processing method according to a second embodiment of the present invention, where this embodiment is applicable to a case of batch processing data, and the method may be executed by a data batch processing system, where the data batch processing system may be implemented by software and/or hardware, and specifically includes the following steps:
s210, converting the original data into a preset data format according to a reading command in the batch processing task, and reading the original data in the preset data format.
S220, processing the original data based on the processing command in the batch processing task, and storing the processed data into a preset list.
And S230, converting the data in the preset list into a second preset data format according to the write command in the batch processing task, and writing the data in the second preset data format into the target storage container.
On the basis of the technical scheme of the embodiment, the method further comprises the following steps:
the batch processing method comprises the steps of receiving batch processing tasks by a task management module, controlling a data reading module, a data processing module and a data writing module to execute the batch processing tasks according to the batch processing tasks, wherein the batch processing tasks comprise original data, a reading command, a processing command and a writing command.
Optionally, based on the task management module, the read command is generated according to the initial storage location of the original data in the batch processing task, the processing command is generated according to the processing requirement in the batch processing task, and the write command is generated according to the target storage location of the data in the batch processing task.
On the basis of the technical scheme of the embodiment, the method further comprises the following steps:
monitoring whether data has abnormal conditions in the write command execution process;
if abnormal conditions appear, then generate alarm information to save alarm information in information memory, wherein, alarm information includes: abnormal data and abnormal reasons, wherein the abnormal reasons comprise at least one of the following items: data format anomalies and network anomalies;
if no abnormal condition exists, generating processing state information of each batch processing task according to the processing state of each batch processing task; wherein the processing state comprises: the untreated, treated and neutralized treatments were completed.
On the basis of the technical scheme of the embodiment, the method further comprises the following steps:
based on the operation starting module, based on the received batch processing task, sending a starting instruction of the batch processing task to the task management module;
and the task management module controls the data reading module, the data processing module and the data writing module to execute batch processing tasks according to the starting instruction.
Optionally, based on the job starting module, when the batch processing task fails to be executed, the batch processing task is restarted according to the execution failure node of the batch processing task.
Optionally, on the basis of the technical solution of the foregoing embodiment, as shown in fig. 3, the method may further set a multithread processing mode, and may perform parallel processing on multiple batch processing tasks, thereby improving the execution efficiency of the system.
According to the technical scheme, when data are processed in batches, reading, processing and writing of original data in a batch processing task are respectively and independently executed, and codes for reading, processing and writing of the data are not written together, so that when one or more of the codes for reading, processing and writing of the original data need to be modified, only the codes which need to be modified, all the codes do not need to be rewritten, and high maintainability and reusability of the codes are realized in a configuration mode; when data is read, converting original data into a data format which can be read by a system according to a read command in the batch processing task, and reading the data; processing the read original data based on a processing command in the batch processing task, and storing the processed data into a preset list; when data is written, data in the preset list is converted into a data format which can be stored in the target storage container according to a write command in the batch processing task, the data is written into the target storage container, the data format of original data is converted in the processing process of the batch processing task, the data can be conveniently read and written, the problem that reading and writing cannot be carried out due to the fact that the data format does not meet the requirements of reading and writing is solved, and the compatibility of a plurality of data formats is improved.
EXAMPLE III
Fig. 4 is a schematic structural diagram of an apparatus according to embodiment 3 of the present invention, as shown in fig. 4, the apparatus includes a processor 30, a memory 31, an input device 32, and an output device 33; the number of the processors 30 in the device may be one or more, and one processor 30 is taken as an example in fig. 4; the processor 30, the memory 31, the input means 32 and the output means 33 in the device may be connected by a bus or other means, as exemplified by the bus connection in fig. 4.
The memory 31 is used as a computer-readable storage medium for storing software programs, computer-executable programs, and modules, such as program instructions/modules (e.g., the data reading module 11, the data processing module 12, and the data writing module 13) corresponding to a data batch processing method in the embodiment of the present invention. The processor 30 executes various functional applications of the device and data processing by running software programs, instructions and modules stored in the memory 31, so as to realize the data batch processing method.
The memory 31 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the memory 31 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, the memory 31 may further include memory located remotely from the processor 30, which may be connected to the device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 32 may be used to receive input numeric or character information and to generate key signal inputs relating to user settings and function control of the apparatus, such as a keyboard and mouse. The output device 33 may include a display device such as a display screen.
Example four
The fourth embodiment of the present invention further provides a storage medium containing computer-executable instructions, which are used for executing a data batch processing method when executed by a computer processor.
Of course, the storage medium provided by the embodiment of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the method operations described above, and may also perform related operations in a data batch processing method provided by any embodiment of the present invention.
From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly, can also be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.
It should be noted that, in the embodiment of the data batch processing apparatus, each included unit and module are only divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A data batch processing system, the system comprising: the device comprises a data reading module, a data processing module and a data writing module, wherein the data reading module, the data processing module and the data writing module are respectively and independently packaged;
the data reading module is used for converting the original data in the batch processing task into a first preset data format and reading the original data in the first preset data format according to a reading command in the batch processing task;
the data processing module is used for processing the original data based on the processing command in the batch processing task and storing the processed data into a preset list;
and the data writing module is used for converting the data in the preset list into a second preset data format according to the writing command in the batch processing task and writing the data in the second preset data format into the target storage container.
2. The system of claim 1, wherein the data batch processing system further comprises a task management module, and the task management module is configured to receive a batch processing task and control the data reading module, the data processing module, and the data writing module to execute the batch processing task according to the batch processing task, wherein the task management module is packaged independently, and the batch processing task includes raw data, a read command, a process command, and a write command.
3. The system of claim 2, wherein the task management module is further configured to:
and generating the read command according to an initial storage position of original data in a batch processing task, generating the processing command according to a processing requirement in the batch processing task, and generating the write command according to a target storage position of data in the batch processing task.
4. The system of claim 1, further comprising: a data monitoring module;
the data monitoring module is respectively in communication connection with the data processing module and the data writing module and is used for monitoring whether the data has abnormal conditions in the write command execution process;
if abnormal conditions occur, alarm information is generated and stored in an information memory, wherein the alarm information comprises: abnormal data and abnormal causes, the abnormal causes including at least one of: the data format exception and the network exception;
if no abnormal condition exists, generating processing state information of each batch processing task according to the processing state of each batch processing task; wherein the processing state comprises: the untreated, treated and neutralized treatments were completed.
5. The system of claim 2, further comprising: an operation starting module;
the job starting module is used for sending a starting instruction of the batch processing task to the task management module based on the received batch processing task;
and the task management module controls the data reading module, the data processing module and the data writing module to execute the batch processing task according to the starting instruction.
6. The system of claim 5, wherein the job initiation module is further configured to restart the batch task according to a failed execution node of the batch task when the batch task fails to execute.
7. A method for batch processing of data, the method comprising:
converting original data into a preset data format according to a read command in the batch processing task, and reading the original data in the preset data format;
processing the original data based on the processing command in the batch processing task, and storing the processed data into a preset list;
and converting the data in the preset list into a second preset data format according to the write command in the batch processing task, and writing the data in the second preset data format into the target storage container.
8. The method of claim 7, further comprising: monitoring whether data has abnormal conditions in the write command execution process;
if abnormal conditions occur, alarm information is generated and stored in an information memory, wherein the alarm information comprises: abnormal data and abnormal causes, the abnormal causes including at least one of: the data format exception and the network exception;
if no abnormal condition exists, generating processing state information of each batch processing task according to the processing state of each batch processing task; wherein the processing state comprises: the untreated, treated and neutralized treatments were completed.
9. An apparatus, characterized in that the apparatus comprises:
one or more processors;
storage means for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a method of data batch processing as claimed in any one of claims 7 to 8.
10. A storage medium containing computer-executable instructions for performing a method of data batch processing as claimed in any one of claims 7 to 8 when executed by a computer processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911198742.9A CN110895490A (en) | 2019-11-29 | 2019-11-29 | Data batch processing system, method, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911198742.9A CN110895490A (en) | 2019-11-29 | 2019-11-29 | Data batch processing system, method, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110895490A true CN110895490A (en) | 2020-03-20 |
Family
ID=69787078
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911198742.9A Pending CN110895490A (en) | 2019-11-29 | 2019-11-29 | Data batch processing system, method, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110895490A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112291212A (en) * | 2020-10-16 | 2021-01-29 | 北京锐安科技有限公司 | Static rule management method and device, electronic equipment and storage medium |
| CN112579683A (en) * | 2020-12-30 | 2021-03-30 | 广州华资软件技术有限公司 | Method for efficiently accessing Tbase data in batches |
| CN112765216A (en) * | 2021-01-19 | 2021-05-07 | 澜途集思生态科技集团有限公司 | Data batch processing method based on Internet of things |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102999537A (en) * | 2011-09-19 | 2013-03-27 | 阿里巴巴集团控股有限公司 | System and method for data migration |
| CN107133111A (en) * | 2017-04-27 | 2017-09-05 | 努比亚技术有限公司 | A kind of message processing apparatus and method |
| CN108520569A (en) * | 2018-03-30 | 2018-09-11 | 深圳市华宝电子科技有限公司 | Storage method, device, system and the storage medium of video data |
| CN109033184A (en) * | 2018-06-27 | 2018-12-18 | 中国建设银行股份有限公司 | Data processing method and device |
| CN110275767A (en) * | 2019-06-26 | 2019-09-24 | 深圳前海微众银行股份有限公司 | A kind of batch data processing method and processing device |
-
2019
- 2019-11-29 CN CN201911198742.9A patent/CN110895490A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102999537A (en) * | 2011-09-19 | 2013-03-27 | 阿里巴巴集团控股有限公司 | System and method for data migration |
| CN107133111A (en) * | 2017-04-27 | 2017-09-05 | 努比亚技术有限公司 | A kind of message processing apparatus and method |
| CN108520569A (en) * | 2018-03-30 | 2018-09-11 | 深圳市华宝电子科技有限公司 | Storage method, device, system and the storage medium of video data |
| CN109033184A (en) * | 2018-06-27 | 2018-12-18 | 中国建设银行股份有限公司 | Data processing method and device |
| CN110275767A (en) * | 2019-06-26 | 2019-09-24 | 深圳前海微众银行股份有限公司 | A kind of batch data processing method and processing device |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112291212A (en) * | 2020-10-16 | 2021-01-29 | 北京锐安科技有限公司 | Static rule management method and device, electronic equipment and storage medium |
| CN112291212B (en) * | 2020-10-16 | 2023-02-28 | 北京锐安科技有限公司 | Static rule management method and device, electronic equipment and storage medium |
| CN112579683A (en) * | 2020-12-30 | 2021-03-30 | 广州华资软件技术有限公司 | Method for efficiently accessing Tbase data in batches |
| CN112765216A (en) * | 2021-01-19 | 2021-05-07 | 澜途集思生态科技集团有限公司 | Data batch processing method based on Internet of things |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8332845B2 (en) | Compile timing based on execution frequency of a procedure | |
| CN110895490A (en) | Data batch processing system, method, equipment and storage medium | |
| US20150100832A1 (en) | Method and system for selecting and executing test scripts | |
| US8782643B2 (en) | Device and method for controlling communication between BIOS and BMC | |
| CN110663006B (en) | Method for performing failover of programmable logic controller and controlling physical system | |
| US20150100830A1 (en) | Method and system for selecting and executing test scripts | |
| US9824229B2 (en) | Controller with enhanced reliability | |
| KR20220055151A (en) | Method for generating identification id, and apparatus implementing the same method | |
| JP6879625B2 (en) | Programmable controller, management device and control system | |
| CN113094125B (en) | Business process processing method, device, server and storage medium | |
| CN114416284A (en) | Distributed operating system control method, apparatus, device, medium, and program product | |
| KR102194974B1 (en) | System for monitoring and controling electric power system for process verification | |
| CN113805925A (en) | Online upgrading method, device, equipment and medium for distributed cluster management software | |
| CN112083947A (en) | Software package issuing method for supply chain multi-language environment | |
| CN111324482A (en) | Computer application program running data fault processing system | |
| CN114791900A (en) | Operator-based Redis operation and maintenance method, device, system and storage medium | |
| CN111435356A (en) | Data feature extraction method and device, computer equipment and storage medium | |
| JP2009116618A (en) | Information processing apparatus | |
| CN113051122B (en) | Performance data acquisition method, device, electronic equipment and medium | |
| CN111581042B (en) | Cluster deployment method, deployment platform and server to be deployed | |
| KR102434126B1 (en) | Apparatus for processing programmable logic controller program | |
| CN112367205A (en) | Processing method and scheduling system for HTTP scheduling request | |
| CN110908805A (en) | Information distribution method, robot and storage medium | |
| CN110647526B (en) | Batch data processing method, device, computer equipment and storage medium | |
| US11941432B2 (en) | Processing system, processing method, higher-level system, lower-level system, higher-level program, and lower-level program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200320 |
|
| RJ01 | Rejection of invention patent application after publication |