CN117762656A - Data processing method and device, electronic equipment and storage medium - Google Patents

Abstract

The application relates to a data processing method and device, electronic equipment and storage medium, wherein the method comprises the following steps: under the condition that a file operation request for executing a synchronous task is received and the file is determined to be configured with a replication rule, synchronous strategy information is acquired from a database, and a replication event is submitted to a first message queue in a message middleware, wherein the replication event comprises a task for representing the synchronous strategy information; distributing the synchronous task to a second message queue of the message middleware, and submitting synchronous task information to a third message queue of the message middleware; and acquiring a synchronous task from the second message queue, and interacting with the remote storage application programming interface API to realize task synchronization. By the method and the device, the problem of high iteration cost caused by the fact that one file writing event corresponds to a plurality of business of different-place synchronous tasks in the prior art is solved.

Description

Data processing method and device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of data processing, and in particular, to a data processing method and apparatus, an electronic device, and a storage medium.

Background

For the use scene of a single object storage system, files cannot be completely guaranteed not to be lost, so that multi-place and multi-cloud modes are commonly used at present to realize multi-copy redundancy storage in different places. The multi-place and multi-cloud storage system and the original system are 2 sets of relatively independent storage systems in different networks, and the API of the remote storage system can be called according to event content and a synchronization strategy to realize remote synchronous operation, namely, one write event corresponds to one remote synchronous task. Because the call of the remote synchronous API can often generate various problems such as network, authority and the like, the implementation often needs to carry out fault-tolerant development in a targeted way so as to repeatedly execute failure tasks after the abnormal situation is processed. In the aspect of managing task progress, for a plurality of remote synchronous tasks, a corresponding progress management flow needs to be established to track and manage the state and progress of each task, meanwhile, management complexity is increased, and monitoring and scheduling are needed to ensure smooth execution of the tasks.

Based on the scheme, the current file remote synchronization scheme uses a certain system as a center, the file operation directly related to the system is recorded into a relational database table as a write event, another process monitors the list, and when the process finds that a new event occurs, the remote synchronization operation is implemented by combining information carried by the event with a remote strategy, but the repeated reading and writing of the table event are involved in the whole process, especially the failure condition occurs, and the repeated reading and writing of the table are more frequent. Therefore, in the scheme in the prior art, one file writing event corresponds to a plurality of business scenes which are synchronized in different places, so that a single business scene is required to realize a whole set of processes of inter-transmission task monitoring, task progress management, fault tolerance and the like, and the iteration cost is high.

Disclosure of Invention

The application provides a data processing method and device, electronic equipment and a storage medium, and aims to solve the problem that iteration cost is high due to the fact that a file writing event corresponds to a plurality of business of different-place synchronous tasks in the prior art.

In a first aspect, the present application provides a method for processing data, including: under the condition that a file operation request for executing a synchronous task is received and the file is determined to be configured with a replication rule, synchronous strategy information is obtained from a database, and a replication event is submitted to a first message queue in a message middleware, wherein the replication event comprises a task for representing the synchronous strategy information; distributing the synchronous task to a second message queue of the message middleware, and submitting synchronous task information to a third message queue of the message middleware, wherein the second message queue is a synchronous task queue, and the third message queue is a task progress management queue; and acquiring the synchronous task from the second message queue, and interacting with a remote storage application programming interface API to realize task synchronization.

In a second aspect, the present application provides a data processing apparatus, including: the first processing module is used for acquiring synchronous strategy information from a database and submitting a copy event to a first message queue in a message middleware under the condition that a file operation request for executing a synchronous task is received and the file is determined to be configured with a copy rule, wherein the copy event comprises a task for representing the synchronous strategy information; the second processing module is used for distributing the synchronous task to a second message queue of the message middleware and submitting synchronous task information to a third message queue of the message middleware, wherein the second message queue is a synchronous task queue, and the third message queue is a task progress management queue; and the third processing module is used for acquiring the synchronous task from the second message queue and interacting with a remote storage Application Programming Interface (API) to realize task synchronization.

In a third aspect, the present application provides an electronic device, including: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor coupled to the at least one bus; at least one memory coupled to the at least one bus, wherein the processor is configured to perform the data processing method of the first aspect of the present application.

In a fourth aspect, the present application further provides a computer storage medium storing computer executable instructions for performing the data processing method according to the first aspect of the present application.

Compared with the prior art, the technical scheme provided by the embodiment of the application has the following advantages: in the embodiment of the application, the task synchronization flow and the task progress management flow are mutually independent through the message queue in the message middleware, namely, the event of the task to be synchronized can be submitted to the first message queue, the task to be synchronized is submitted to the second message queue when the task to be synchronized is synchronized, and the synchronous task information is submitted to the third message queue, so that the mutual independence of each flow is realized, the structural repeated design is avoided, the cost of data iteration is reduced, and the problem that the iteration cost is high due to the fact that one file write event corresponds to a plurality of businesses of different-place synchronous tasks in the prior art is solved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.

In order to more clearly illustrate the embodiments of the invention or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, and it will be obvious to a person skilled in the art that other drawings can be obtained from these drawings without inventive effort.

One or more embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like references indicate similar elements, and in which the figures of the drawings are not to be taken in a limiting sense, unless otherwise indicated.

Fig. 1 is a flowchart of a data processing method according to an embodiment of the present application;

FIG. 2 is one of alternative flowcharts of a method for processing data according to an embodiment of the present application;

FIG. 3 is a second flowchart of an alternative method for processing data according to an embodiment of the present disclosure;

fig. 4 is a schematic structural diagram of a data processing device according to an embodiment of the present application;

fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present application based on the embodiments herein.

The following disclosure provides many different embodiments, or examples, for implementing different structures of the invention. In order to simplify the present disclosure, components and arrangements of specific examples are described below. They are, of course, merely examples and are not intended to limit the invention. Furthermore, the present invention may repeat reference numerals and/or letters in the various examples. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed.

Firstly, related terms in the embodiments of the present application are explained;

public cloud: and the third party cloud service provider provides services such as computing resources, storage space, application programs and the like for a large number of users through a public network.

Private cloud: cloud infrastructure, built and managed by the enterprise or organization itself, is used to meet its internal needs. Private clouds offer flexibility and automation characteristics similar to public clouds, but resources and services are only available for internal use and not external.

Hybrid cloud service: the method combines two different types of cloud computing environments, namely public cloud and private cloud.

Object data storage: a way of storing files of unstructured data.

Cloudy environment: using a plurality of different cloud service providers.

K-V database: a database model for storing data based on key value pairs. In a K-V database, data is stored and retrieved in the form of Key-Value pairs, where a Key (Key) is a unique identifier and a Value (Value) is the data associated with the Key.

Relational database: a relational model based database management system (Database Management System, DBMS). In a relational database, data is organized in the form of tables, which are made up of rows and columns, where each row represents an entity record and each column represents an attribute in the record.

Fig. 1 is a flowchart of a data processing method according to an embodiment of the present application, where, as shown in fig. 1, the method includes the steps of:

step 101, under the condition that a file operation request for executing a synchronous task is received and the file is determined to be configured with a replication rule, synchronous strategy information is obtained from a database and a replication event is submitted to a first message queue in a message middleware, wherein the replication event comprises a task for representing the synchronous strategy information;

in a specific example, the synchronization policy refers to synchronizing the current task to the corresponding one or more destinations, e.g., the current target task is invoked by destination a and destination B simultaneously, and the synchronization policy refers to synchronizing the target task to destination a and destination B. In addition, the message middleware in the embodiment of the application can include a plurality of message queues, and each message queue has a corresponding function. The database in the embodiments of the present application may be a K-V database, or other relational database.

Step 102, distributing the synchronous task to a second message queue of the message middleware, and submitting synchronous task information to a third message queue of the message middleware, wherein the second message queue is a synchronous task queue, and the third message queue is a task progress management queue;

step 103, obtain the synchronous task from the second message queue, and interact with the remote storage application programming interface (Application Programming Interface, API) to achieve task synchronization.

Through the steps 101 to 103, in the embodiment of the present application, the task synchronization process and the task progress management process are mutually independent through the message queue in the message middleware, that is, the event of the task to be synchronized is submitted to the first message queue, the task to be synchronized is submitted to the second message queue when the task to be synchronized is synchronized, and the synchronization task information is submitted to the third message queue, so that each process is mutually independent, structural repetitive design is avoided, the cost of data iteration is reduced, and the problem that in the prior art, the iteration cost is higher because of the service of the task to be synchronized corresponding to a plurality of different places for one file writing event is solved.

In an optional implementation manner of the embodiment of the present application, the method of the embodiment of the present application may further include:

step 11, after the synchronous task is obtained from the second message queue, the synchronous task is stored in the database.

It can be seen that, in this embodiment of the present application, the synchronization task may be stored in a database, such as a K-V database, when the synchronization task is acquired from the second message queue, so as to make the synchronization task persistent, and if the synchronization task is abnormal later, the synchronization task may be acquired from the database again, and submitted to the second message queue.

In an alternative implementation manner of the embodiment of the present application, the method of the embodiment of the present application, as shown in fig. 2, may further include:

step 201, deleting task records of the synchronous tasks from the database after the task synchronization is completed, and submitting the results to a third message queue, wherein the results indicate that the task synchronization is completed and the synchronous tasks are deleted, and the third message queue comprises dotting data indicating the progress of the tasks.

In a specific example, the third message queue in the embodiment of the present application is dotting data for storing the task progress, and the dotting data is data for recording the task progress, so that the execution state of each task at present can be obtained through the dotting data in the third message queue. In other words, in the embodiment of the present application, the task progress management may also be implemented through a separate message queue, which is independent of the execution process of the synchronous task.

In an alternative implementation manner of the embodiment of the present application, as shown in fig. 2, the method in the embodiment of the present application may further include:

step 202, obtaining dotting data from a third message queue, and storing the currently obtained dotting data;

and 203, comparing the currently acquired dotting data with the dotting data stored before, aggregating the dotting data based on the comparison result, and submitting the aggregated dotting data to a fourth message queue of the message middleware.

For the above steps 202 and 203, in a specific example, the dotting data stored as before is: bucket-rule: 2, execution completion 1, wherein 2 indicates that there are two files, and execution completion 1 indicates that file 1 has been executed. If there is currently also File 3 on the basis of File 1 and File 2, then the currently acquired dotting data may be bucket-rule: 2, performing to finish 1, +1, wherein +1 represents the file 3, specifically may represent the execution state of the file 3, and then the current acquired dotting data and the previous acquired dotting data need to be aggregated, and the aggregated result is submitted to a fourth message queue.

On the basis of the data processing method executed by the embodiment of the present application in fig. 1, as shown in fig. 2, the method of the embodiment of the present application may further include:

step 204, merging the aggregated dotting data obtained from the fourth message queue with the corresponding data in the database;

step 205, updating the combined result into the database.

Through the above steps 204 and 205, in the embodiment of the present application, the task progress may be saved to the database, so that the progress result may be directly obtained from the database to reduce the calculation at the time of obtaining.

On the basis of the data processing method executed by the embodiment of the present application in fig. 1, as shown in fig. 3, the method in the embodiment of the present application may further include:

step 301, under the condition that the synchronization abnormality occurs in the task synchronization process, submitting the synchronization task with the synchronization abnormality to a fifth message queue in the message middleware;

step 302, if the submission to the fifth message queue fails, the synchronization task with the synchronization exception is submitted to the database.

Therefore, in the embodiment of the application, the task with the abnormal synchronization can be submitted to the fifth message queue, or if the task fails to be submitted to the fifth message queue, the task can be directly submitted to the database to ensure that the data of the task is not lost, thereby improving the safety of the data of the task.

In an optional implementation manner of the embodiment of the present application, the method in the embodiment of the present application may further include:

step 21, scanning a table in a database; the table is used for recording information in the task synchronization process;

step 22, under the condition that the first record is scanned, submitting the task corresponding to the first record to a second message queue, wherein the first record is used for recording the lost record of the synchronous task in the second message queue in the task synchronization process;

and step 23, under the condition that the second record is scanned, submitting the task corresponding to the second record to a sixth message queue of the message middleware, wherein the second record is used for recording the task record of which the message queue in the message middleware fails to be input.

In the embodiment of the application, when the synchronous task is abnormal, self-healing fault tolerance can be realized through the modes from the step 21 to the step 23, namely, in the embodiment of the application, abnormal data can be centrally managed through the message middleware, self-healing compensation can be uniformly performed, and the efficiency of data compensation is improved.

The following explains the manner in the embodiment of the present application by taking the database as a K-V database as an example, and in this specific embodiment, the synchronization process will be refined, that is, the process is split into a file replication process, a task progress management process, and a self-healing fault tolerance process. The file copying means that the copying task is generated and the task file is synchronously ended; the task number progress management flow refers to synchronous progress management under each rule; the self-healing fault-tolerant flow is to monitor failure conditions and automatically recover.

1) The file copying process comprises the following steps: the method comprises the steps of distributing synchronous tasks of synchronous task production events and synchronous events in message middleware to a task queue to be synchronized, processing the message middleware synchronous task queue, consuming file memory task queues and carrying out abnormal processing on the task middle.

The production event for the synchronous task refers to that after the API receives a file operation request, if the file is configured with a copying rule, a copying event is submitted to the message middleware. The specific flow comprises the following steps: and acquiring synchronization strategy information from the K-V database, determining whether the event needs to be synchronized in different places, and submitting the event to a message middleware queue if the event needs to be synchronized in different places.

The process of distributing the synchronous tasks of the synchronous events in the message middleware to the task queues to be synchronized comprises the following steps: and determining that tasks need to be distributed to corresponding task queues to be synchronized through a K-V database or a local cache, submitting synchronous task information in an initial state to a progress queue for dotting (namely recording mark remaining), and waiting for task number progress management processing.

The flow of processing the message middleware synchronous task queue comprises the following steps: and acquiring a synchronous task from the message queue, and ensuring that the fetched data is not lost through double-layer storage of the memory and the K-V database.

The file memory task queue consumption process comprises the following steps: and the memory queue synchronization task is taken out and interacted with the remote storage API to realize synchronization. After the task is completed, deleting the task record corresponding to the K-V database, pushing the result to the message queue, and waiting for task number progress management processing. If the task is abnormal in the middle, the abnormality is submitted to an abnormal message queue, and the self-healing fault-tolerant flow is waited for processing.

The process of exception handling in the middle of the task comprises the following steps: submitting the exception to an exception message queue, and waiting for the self-healing fault-tolerant flow to be processed; if the commit message queue fails, commit to the high availability K-V database, awaiting compensation by the self-healing mechanism.

2) The task number progress management process comprises the following steps: task progress processing, task progress value saving and task progress query.

The task progress processing flow comprises the following steps: consuming dotting data from the message queue, and storing K-V data; the validity of the dotting data is determined by comparing the changes before and after storage, so that a corresponding count value in a state is obtained, dimension aggregation is improved, and the aggregated data is submitted to a message queue.

The task progress value saving process comprises the following steps: and (3) consuming a message queue of the aggregated data, acquiring a corresponding dimension summary result from the K-V database, merging the dimension summary result with the current counting result, and updating the merged value into the K-V database.

The task progress query process comprises the following steps: when the client submits a query request, a unique key value is obtained according to the query condition, and the K-V database is queried by using the key value to obtain a progress index value.

3) The self-healing fault-tolerant flow comprises the following steps: and compensating data loss of a memory queue in an abnormal scene in a file copying process, and compensating failure of recording a message queue in a file copying process and a task number progress management process.

The process of compensating the data loss of the memory queue in the abnormal scene in the file copying process comprises the following steps: and (3) carrying out timing scanning on the related table of the K-V database, and if records are found, submitting the contents to a task queue to be synchronized in a file copying process.

The process of compensating the message queue input failure in the file copying process and the task number progress management process comprises the following steps: and carrying out timing scanning on the K-V database correlation table, and if the content is found, submitting the content to a message queue appointed in the content.

Therefore, in the embodiment of the application, the file copying process is unpacked through multi-step fine granularity, so that the expansion capability of the system is greatly improved, when the strategy needs to be expanded, for example, a cloud manufacturer is newly added in a multi-cloud environment, part of files need to be synchronized to the manufacturer, at the moment, the related functions of the newly added cloud manufacturer can be realized only by carrying out strategy configuration and calling related logic by the API.

In addition, as can be seen from the above manner in the embodiment of the present application, the design of highly decoupling between the abnormal event and the main service is performed, and the abnormal event is uniformly managed, so that the newly added abnormality only needs to be submitted to the abnormal event message queue, and it is determined to which step of formal flow the retry step belongs, and then the self-healing flow resubmisses the failure data to the message queue of the corresponding flow, thereby improving the fault tolerance capability of the system.

In addition, task progress management in the embodiment of the present application is also managed by a separate message queue, and is in communication with the main flow Cheng Jieou. Based on the above, when a new synchronization strategy is generated, the design of a progress acquisition flow is not needed any more, and the flow can be directly multiplexed, namely, only the new aggregation type is needed, and the independent development of a progress management flow is not needed any more, so that the efficiency of acquiring a progress result by a user is improved, and meanwhile, the input of hardware resources can be reduced while the accuracy is ensured.

Corresponding to fig. 1, the embodiment of the present application further provides a data processing device, as shown in fig. 4, where the device includes:

a first processing module 402, configured to, when a file operation request for executing a synchronization task is received and it is determined that the file is configured with a replication rule, acquire synchronization policy information from a database and submit a replication event to a first message queue in a message middleware, where the replication event includes a task that characterizes the synchronization policy information;

the second processing module 404 is configured to distribute the synchronization task to a second message queue of the message middleware, and submit the synchronization task information to a third message queue of the message middleware, where the second message queue is a synchronization task queue, and the third message queue is a task progress management queue;

and a third processing module 406, configured to obtain the task from the second message queue, and interact with the remote storage application programming interface API to achieve task synchronization.

Through the device of the embodiment of the application, the task synchronization flow and the task progress management flow are mutually independent through the message queue in the message middleware, so that the event of the task to be synchronized can be submitted to the first message queue, the task to be synchronized is submitted to the second message queue when the task to be synchronized is synchronized, and the synchronous task information is submitted to the third message queue, thereby realizing the mutual independence of each flow, avoiding structural repeated design, reducing the cost of data iteration, and solving the problem of higher iteration cost caused by the fact that one file write event corresponds to a plurality of businesses of different-place synchronous tasks in the prior art.

In an optional implementation manner of the embodiment of the present application, the apparatus of the embodiment of the present application may further include: and the fourth processing module is used for storing the synchronous task in the database after acquiring the synchronous task from the second message queue.

In an optional implementation manner of the embodiment of the present application, the apparatus of the embodiment of the present application may further include: and the fifth processing module is used for deleting the task record of the synchronous task from the database after the task synchronization is completed, and submitting the result to a third message queue, wherein the result represents that the task synchronization is completed and the synchronous task is deleted, and the third message queue comprises dotting data representing the progress of the task.

In an optional implementation manner of the embodiment of the present application, the apparatus of the embodiment of the present application may further include: the sixth processing module is used for acquiring dotting data from the third message queue and storing the currently acquired dotting data; and the seventh processing module is used for comparing the currently acquired dotting data with the dotting data stored before, aggregating the dotting data based on the comparison result, and submitting the aggregated dotting data to a fourth message queue of the message middleware.

In an optional implementation manner of the embodiment of the present application, the apparatus of the embodiment of the present application may further include: the eighth processing module is used for merging the aggregated dotting data acquired from the fourth message queue with the corresponding data in the database; and the updating module is used for updating the combined result into the database.

In an optional implementation manner of the embodiment of the present application, the apparatus of the embodiment of the present application may further include: the first submitting module is used for submitting the synchronous task with the abnormal synchronization to a fifth message queue in the message middleware under the condition that the abnormal synchronization occurs in the task synchronization process; and the second submitting module is used for submitting the synchronous tasks with abnormal synchronization to the database if the submitting to the fifth message queue fails.

In an optional implementation manner of the embodiment of the present application, the apparatus of the embodiment of the present application may further include: the scanning module is used for scanning a table in the database, wherein the table is used for recording information in the task synchronization process; the ninth processing module is used for submitting the task corresponding to the first record to the second message queue under the condition that the first record is scanned, wherein the first record is used for recording the lost record of the synchronous task in the second message queue in the task synchronization process; and the tenth processing module is used for submitting the task corresponding to the second record to a sixth message queue of the message middleware under the condition that the second record is scanned, wherein the second record is used for recording the task record of which the message queue in the message middleware fails to be input.

As shown in fig. 5, the embodiment of the present application provides an electronic device, which includes a processor 511, a communication interface 512, a memory 513, and a communication bus 514, wherein the processor 511, the communication interface 512, and the memory 513 perform communication with each other through the communication bus 514,

a memory 513 for storing a computer program;

in one embodiment of the present application, the processor 511 is configured to implement the method for processing data provided in any of the foregoing method embodiments when executing the program stored in the memory 513, and the functions of the method are similar, and are not described herein again.

The present application also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the data processing method provided by any one of the method embodiments described above.

The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

From the above description of embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus a general purpose hardware platform, or may be implemented by hardware. Based on such understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the related art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the method described in the respective embodiments or some parts of the embodiments.

It is to be understood that the terminology used herein is for the purpose of describing particular example embodiments only, and is not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises," "comprising," "includes," "including," and "having" are inclusive and therefore specify the presence of stated features, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring their performance in the particular order described or illustrated, unless an order of performance is explicitly stated. It should also be appreciated that additional or alternative steps may be used.

The foregoing is only a specific embodiment of the invention to enable those skilled in the art to understand or practice the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A method of processing data, comprising:

under the condition that a file operation request for executing a synchronous task is received and the file is determined to be configured with a replication rule, synchronous strategy information is obtained from a database, and a replication event is submitted to a first message queue in a message middleware, wherein the replication event comprises a task for representing the synchronous strategy information;

distributing the synchronous task to a second message queue of the message middleware, and submitting synchronous task information to a third message queue of the message middleware, wherein the second message queue is a synchronous task queue, and the third message queue is a task progress management queue;

and acquiring the synchronous task from the second message queue, and interacting with a remote storage application programming interface API to realize task synchronization.

2. The method according to claim 1, wherein the method further comprises:

and after the synchronous task is acquired from the second message queue, storing the synchronous task in the database.

3. The method according to claim 1, wherein the method further comprises:

and deleting the task record of the synchronous task from the database after the task is synchronously completed, and submitting a result to the third message queue, wherein the result represents that the task is synchronously completed and the synchronous task is deleted, and the third message queue comprises dotting data representing the progress of the task.

4. A method according to claim 3, characterized in that the method further comprises:

obtaining dotting data from the third message queue, and storing the currently obtained dotting data;

comparing the currently acquired dotting data with the dotting data stored before, aggregating the dotting data based on the comparison result, and submitting the aggregated dotting data to a fourth message queue of the message middleware.

5. The method according to claim 4, wherein the method further comprises:

merging the aggregated dotting data acquired from the fourth message queue with corresponding data in the database;

and updating the combined result into the database.

6. The method according to claim 1, wherein the method further comprises:

under the condition that the synchronization abnormality occurs in the task synchronization process, submitting the synchronization task with the synchronization abnormality to a fifth message queue in the message middleware;

if the submission to the fifth message queue fails, submitting the synchronous task with abnormal synchronization to the database.

7. The method according to claim 1, wherein the method further comprises:

scanning a table in the database, wherein the table is used for recording information in a task synchronization process;

under the condition that a first record is scanned, submitting a task corresponding to the first record to the second message queue, wherein the first record is used for recording a lost record of a synchronous task in the second message queue in the task synchronization process;

and under the condition that a second record is scanned, submitting a task corresponding to the second record to a sixth message queue of the message middleware, wherein the second record is used for recording a task record of which the message queue in the message middleware fails to be input.

8. A data processing apparatus, comprising:

the first processing module is used for acquiring synchronous strategy information from a database and submitting a copy event to a first message queue in a message middleware under the condition that a file operation request for executing a synchronous task is received and the file is determined to be configured with a copy rule, wherein the copy event comprises a task for representing the synchronous strategy information;

the second processing module is used for distributing the synchronous task to a second message queue of the message middleware and submitting synchronous task information to a third message queue of the message middleware, wherein the second message queue is a synchronous task queue, and the third message queue is a task progress management queue;

and the third processing module is used for acquiring the synchronous task from the second message queue and interacting with a remote storage Application Programming Interface (API) to realize task synchronization.

9. An electronic device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor coupled to the at least one bus; at least one memory coupled to the at least one bus, wherein the processor is configured to perform the method of processing data according to any of the claims 1 to 7.

10. A computer storage medium storing computer executable instructions for performing the method of processing data according to any one of claims 1 to 7 of the present application.