CN114579059A

CN114579059A - File data migration method, device, equipment and storage medium

Info

Publication number: CN114579059A
Application number: CN202210245300.0A
Authority: CN
Inventors: 钟丽波; 鲁莎莎
Original assignee: Shanghai Eisoo Information Technology Co Ltd
Current assignee: Shanghai Eisoo Information Technology Co Ltd
Priority date: 2022-03-14
Filing date: 2022-03-14
Publication date: 2022-06-03

Abstract

The invention discloses a file data migration method, a file data migration device, file data migration equipment and a storage medium. Scanning each file item in the task to be migrated to obtain stock scanning information by acquiring the task to be migrated, and writing the stock scanning information into a scanning list; acquiring a target file item operation executed by a task to be migrated after scanning starts, determining incremental scanning information according to the target file item operation, and writing the incremental scanning information into a scanning list; migrating the file items to be migrated corresponding to the scanning information in the scanning list to a target file storage system, wherein the scanning information comprises: the storage scanning information and the increment scanning information can acquire and record the increment change of the file data in the scanning process in real time and synchronize the increment change to the scanning list, repeated scanning is not needed, the consistency of file data migration can be ensured, the migration complexity is reduced, and the migration efficiency is improved.

Description

File data migration method, device, equipment and storage medium

Technical Field

The present invention relates to the field of data migration technologies, and in particular, to a file data migration method, apparatus, device, and storage medium.

Background

With the advent of the big data era, unstructured data of enterprises are rapidly increased, and in order to better manage the data and enhance the maintainability of the data, massive files in an original file storage system need to be migrated to an unstructured data management system for unified management.

In the data migration process, it is necessary to ensure that the storage service required by the service system cannot be interrupted, and in the prior art, generally, an NAS gateway system performs online migration of file data while accepting a service IO. The main process is as follows: and scanning data of the original storage system, then carrying out file data migration according to the scanning result, and simultaneously recording an incremental log. After the historical data migration is finished, performing incremental file data migration according to the log in sequence.

Since the historical data may be massive, and the scanning generally adopts a depth or breadth traversal mode to gradually record the full path of the data in the original storage system, the scanning may need to be continued for a while. If there are renaming or moving operations on the scanned historical file data before the scanning is completely finished, there is a possibility that the scanning item cannot be found in the original storage when the migration is performed according to the scanning result. Once the situation is met, the original storage system needs to be subjected to full scan comparison for multiple times to ensure the consistency of file data in the target migration device and the original storage device, so that the migration complexity is high, the efficiency is low, a large amount of computing resources are occupied, and even the performance of the service system may be affected.

Disclosure of Invention

The invention provides a file data migration method, a file data migration device, file data migration equipment and a storage medium, which are used for solving the problems that in the prior art, file data change in the scanning process requires multiple full scanning comparisons, and can acquire and record incremental changes of the file data in the scanning process in real time, so that the consistency of file data migration can be ensured without repeated scanning, the migration complexity is reduced, and the migration efficiency is improved.

According to an aspect of the present invention, a file data migration method is provided, which is applied to an original file storage system, and the method includes:

acquiring a task to be migrated, scanning each file item in the task to be migrated to obtain stock scanning information, and writing the stock scanning information into a scanning list;

acquiring a target file item operation executed on the task to be migrated after scanning starts, determining incremental scanning information according to the target file item operation, and writing the incremental scanning information into a scanning list;

migrating the file items to be migrated corresponding to the scanning information in the scanning list to a target file storage system, wherein the scanning information comprises: the inventory scan information and the incremental scan information.

According to another aspect of the present invention, there is provided a file data migration apparatus configured to be disposed in an original file storage system, the apparatus including:

the stock scanning module is used for acquiring a task to be migrated, scanning each file item in the task to be migrated to obtain stock scanning information, and writing the stock scanning information into a scanning list;

the incremental scanning module is used for acquiring a target file item operation executed on the task to be migrated after scanning starts, determining incremental scanning information according to the target file item operation, and writing the incremental scanning information into a scanning list;

a migration module, configured to migrate the file items to be migrated, which correspond to the respective scanning information in the scanning list, to a target file storage system, where the scanning information includes: the inventory scan information and the incremental scan information.

According to another aspect of the present invention, there is provided a computer apparatus comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores a computer program executable by the at least one processor and an original file storage system, the computer program being executable by the at least one processor to enable the at least one processor to perform the file data migration method according to any embodiment of the present invention.

According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the file data migration method according to any one of the embodiments of the present invention when the computer instructions are executed.

According to the technical scheme of the embodiment of the invention, the stock scanning information is obtained by acquiring the task to be migrated and scanning each file item in the task to be migrated, and the stock scanning information is written into the scanning list; acquiring a target file item operation executed by a task to be migrated after scanning starts, determining incremental scanning information according to the target file item operation, and writing the incremental scanning information into a scanning list; migrating the file items to be migrated corresponding to the scanning information in the scanning list to a target file storage system, wherein the scanning information comprises: the method and the device have the advantages that the problems that in the prior art, when the file item is subjected to target operation in the scanning process and is migrated, the scanning item cannot be found in the original storage and repeated full scanning comparison is needed are solved, the incremental change of the file data in the scanning process can be obtained and recorded in real time, a scanning list is modified, repeated scanning is not needed, the consistency of file data migration can be guaranteed, the migration complexity is reduced, and the migration efficiency is improved.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present invention, nor do they necessarily limit the scope of the invention. Other features of the present invention will become apparent from the following description.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 is a flowchart of a file data migration method according to an embodiment of the present invention;

FIG. 2 is a flowchart of a file data migration method according to a second embodiment of the present invention;

FIG. 3 is a flowchart of a file data migration method according to a third embodiment of the present invention;

FIG. 4 is a flowchart of a file data migration method according to a fourth embodiment of the present invention;

FIG. 5 is a flowchart of a file data migration method according to a fifth embodiment of the present invention;

fig. 6 is a schematic structural diagram of a file data migration apparatus according to a sixth embodiment of the present invention;

fig. 7 is a schematic structural diagram of a computer device implementing the file data migration method according to an embodiment of the present invention.

Detailed Description

In order to make those skilled in the art better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Example one

Fig. 1 is a flowchart of a file data migration method according to an embodiment of the present invention, where this embodiment is applicable to a case where file data in an original file storage system is migrated to a target file storage system, and the method may be executed by a file data migration apparatus, where the apparatus may be integrated in the original file storage system, and the original file storage system and the target file storage system may be configured in the same computer device, or may be configured in two different computer devices, respectively. As shown in fig. 1, the method includes:

s110, acquiring a task to be migrated, scanning each file item in the task to be migrated to obtain stock scanning information, and writing the stock scanning information into a scanning list.

The task to be migrated is the sum of all file items which need to be migrated to the target file storage system in the original file storage system. The file items can be general names of files and directories, the directories can be regarded as folders, and multi-level directories and files can be included under the directories; the file contains file data, but no files or directories are contained under the file.

Because the attribute information of the file items may change at any time in the scanning process, the scanning information corresponding to each file item of the task to be migrated, which is stored in the original file storage system, is called stock scanning information; correspondingly, the scanning information corresponding to the file item with the changed or added attribute information after the scanning is started is called incremental scanning information. The scan information may be information such as the name, index number, and path of the file entry. The scan list is a list for storing scan information, and file data scanned from an original storage system is migrated to a target file storage system according to the scan list.

For example, the manner of acquiring the task to be migrated by the original file storage system may be to receive a migration instruction input by a user in the original file storage system, and determine the task to be migrated according to the migration instruction. The migration instructions may include information such as the migration task and the target file storage system to which the migration task is migrated.

S120, acquiring a target file item operation executed by the task to be migrated after the scanning starts, determining incremental scanning information according to the target file item operation, and writing the incremental scanning information into a scanning list.

The incremental scanning information refers to scanning information corresponding to each file item, wherein the scanning information corresponds to the attribute information of each file item, which is changed or newly added due to the operation of the target file item in the task to be migrated after the scanning starts. The target file item operation may be, for example, a file item creation operation or a file item renaming operation, etc.

Specifically, before the scanning starts, the attribute information of the file item changed or newly added by the operation on the target file item executed in the task to be migrated is scanned and included in the stock scanning information in the scanning process, that is, the incremental scanning information is not generated. And after the scanning starts, the attribute information changed or newly added by the target file item operation executed by the task to be migrated is not included in the stock scanning information, that is, incremental scanning information is generated, and the incremental scanning information can be determined according to the target file item operation executed by the task to be migrated. And writing the incremental scanning information into a scanning list, so that the scanning list comprises the stock scanning information and the incremental scanning information.

S130, migrating the file items to be migrated corresponding to the scanning information in the scanning list to a target file storage system, wherein the scanning information comprises: inventory scan information and incremental scan information.

Wherein the target file storage system may include: an unstructured data management system.

Specifically, a storage path of a file item to be migrated is determined according to each piece of scanning information in the scanning list, and the file item to be migrated is migrated to the target file storage system according to the storage path, where the scanning information includes: scanning the stock scanning information obtained by scanning each file item in the task to be migrated, which also comprises the following steps: incremental scan information determined from target file item operations performed after the scan begins. The file items to be migrated corresponding to the scanning information in the scanning list are migrated to the target file storage system, the file items with the changed attribute information can be migrated to the target file storage system, multiple scanning comparison on the file items in the original file storage system is not needed, the consistency of the migrated data can be ensured, and the migration time and the migration complexity are greatly shortened.

According to the technical scheme of the embodiment of the invention, the stock scanning information is obtained by acquiring the task to be migrated and scanning each file item in the task to be migrated, and the stock scanning information is written into the scanning list; acquiring a target file item operation executed by a task to be migrated after scanning starts, determining incremental scanning information according to the target file item operation, and writing the incremental scanning information into a scanning list; migrating the file items to be migrated corresponding to the scanning information in the scanning list to a target file storage system, wherein the scanning information comprises: the storage scanning information and the increment scanning information can acquire and record the increment change of the file data in the scanning process in real time, repeated scanning is not needed, the consistency of file data migration can be ensured, the migration complexity is reduced, and the migration efficiency is improved.

Example two

Fig. 2 is a flowchart of a file data migration method according to a second embodiment of the present invention, and this embodiment further details the technical feature "scanning each file item in the task to be migrated to obtain the stock scanning information" in step S110 on the basis of the above embodiment. As shown in fig. 2, the method includes:

s210, acquiring the task to be migrated, writing the index numbers of all levels of directories of the task to be migrated into the storage to-be-scanned queue respectively, and reading all current levels of directories in the storage to-be-scanned queue through the scanning thread.

The queue to be scanned is a queue for writing index numbers of the directories in each level in the inventory scanning process, the queue is a special linear table, and similar to a stack, only deletion operation is allowed to be performed at the front end of the queue, and insertion operation is performed at the rear end of the queue.

The current-level directory in the inventory to-be-scanned queue refers to all directories written into the inventory to-be-scanned queue, and the directories belong to the same-level directory of the task to be migrated. The scanning threads can execute tasks of reading the current-level directory in the inventory to be scanned in parallel, and the maximum number of the tasks executed by each scanning thread can be configured according to actual requirements and hardware resource states.

Specifically, the task to be migrated includes multiple levels of file items, each file item corresponds to an index number, and if the type of the file item is a file, the file does not include other file items, and the file does not need to be scanned; if the type of the file entry is a directory, the directory may contain a file or a next-level directory, and the directory needs to be scanned to obtain a next-level file entry of the directory. Therefore, for each level of file items with the types of directories, each level of directory is written into the stock quantity scanning queue respectively. And concurrently executing and reading each current-level directory in the inventory to be scanned through the scanning thread.

For example, the specific writing manner may be a manner of writing the index numbers of the directories at different levels of the task to be migrated into the stock to-be-scanned queue respectively, or a manner of writing the directories at the same level into one stock scanning queue respectively; the index number of the same level of directory can also be written into the stock scanning queue from top to bottom according to the tree structure corresponding to the directory of the task to be migrated, and the specific implementation process is to write the index number of the first level of directory of the task to be migrated into the stock scanning queue, and after the index number of the first level of directory in the stock scanning queue is read, write the index number of the next level of directory into the stock scanning queue until the tree structure corresponding to the directory of the task to be migrated is traversed.

S220, aiming at each current-level directory, scanning the current-level directory to obtain the attribute information of the current stock file item contained in the current-level directory.

The current stock file item refers to each file item contained in the current-level directory of the task to be migrated in the original file storage system before the scanning starts. The current inventory file item includes the current inventory file under the current level directory and/or a next level directory of the current level directory. For example, if the current-level directory is a first-level directory, the file entries contained in the first-level directory are first-level stock file entries.

Specifically, the current-level directory is scanned, all current stock file items included in the current-level directory can be obtained, and attribute information of each current stock file item is obtained.

Optionally, the attribute information of the current stock file item at least includes: the index number of the file entry, the name of the file entry, and the parent directory index number of the file entry.

In a file storage system, file data is stored in data blocks, metadata information describing data attributes is stored in index nodes, each index node has a globally unique index number, and the index number of a file item is used as a unique identification code for identifying the file item. The storage structure of each file item in the task to be migrated can adopt a tree structure, so that for each file item, a parent file item (namely a parent directory) in the tree structure can be queried through the index number of the file item, and the storage path of the file item in the original file storage system is obtained. The attribute information of a file entry also includes the name of the file entry, and generally, file entries in the same directory are not allowed to adopt the same name.

And S230, if the current-level directory is the primary directory, determining stock scanning information of the primary directory according to the attribute information of the primary stock file items contained in the primary directory.

Wherein, the first-level stock file item contained in the first-level catalog comprises: the next directory of the primary directory (i.e., the secondary inventory directory) and the inventory files stored under the primary directory.

Specifically, the primary directory is a top storage location for storing the task to be migrated, and therefore, a storage location needs to be pre-established or selected in the target file storage system to store the task to be migrated, that is, the primary directory of the task to be migrated already exists in the target file storage system. The stock scanning information of the primary catalog is determined only according to the attribute information of the primary stock file item contained in the primary catalog to complete the task to be migrated, and the stock scanning information of the primary catalog specifically includes: the index number and path of the primary inventory file item contained in the primary catalog.

For example, the manner of determining the stock scan information of the primary catalog according to the attribute information of the primary stock file item may be: and determining the name and the path of the stock file item contained in the primary directory according to the index number and the name of the primary stock file item and the index number of the parent directory of the primary stock file item (namely the index number of the primary directory). The path of the stock file items contained in the primary directory is as follows: primary directory index number-the index number of the stock file item contained in the primary directory.

For example, the path of the inventory file items may be determined by: and searching upwards along the directory tree structure of the migration task according to the index number of the stock file item until the index number of the first-level directory is obtained, and determining the path of the stock file item according to the index numbers of the directories at different levels and the index number of the stock file item obtained by searching.

S240, if the current-level directory is not the first-level directory, judging whether the path of the current-level directory exists in the target file storage system, and obtaining stock scanning information of the current-level directory according to the judgment result and the attribute information of the current stock file item.

Specifically, if the current-level directory is not the first-level directory, it is not determined whether the current directory exists in the target file storage system, and therefore, it is necessary to first determine whether the path of the current-level directory exists in the target file storage system. The judgment result may be presence or absence, and the stock scan information of the current-level directory is obtained from the attribute information of the current-level file item based on the judgment result.

Illustratively, if the judgment result is that the path of the previous-level directory exists in the target file storage system, and the current-level directory already exists in the target file storage system, the current-level directory does not need to be migrated, and the inventory scanning information does not need to include the scanning information of the current-level directory, but only includes the scanning information of the current inventory file item under the current-level directory; if the judgment result shows that the current-level directory does not exist in the target file storage system, the current-level directory needs to be migrated, and the stock scanning information needs to include the scanning information of the current-level directory and the scanning information of the current stock file item under the current-level directory.

For example, the path of the current level directory may be as follows: and searching the path upwards along the directory tree structure of the migration task according to the index number of the current-level directory until the index number of the first-level directory is obtained, and determining the path of the current-level directory according to the index numbers of the searched directories.

In a specific example, if the current-level directory is a third-level directory and the index number of the current-level directory is a third index number, the parent path is searched upward to obtain a second index number of the second-level parent directory corresponding to the current-level directory, and then the parent path is searched upward to obtain the index number of the third-level parent directory corresponding to the second-level parent directory, then the path of the current-level directory may be: third index number-second index number-first index number.

And S250, writing the storage scanning information into a scanning list.

Specifically, the stock scanning information corresponding to each level of directory is written into the scanning list, so that the scanning list includes the scanning information of the directories and the file items under the directories, and the file items to be migrated can be migrated according to the stock scanning information in the scanning list.

S260, acquiring a target file item operation executed by the task to be migrated after the scanning starts, determining incremental scanning information according to the target file item operation, and writing the incremental scanning information into a scanning list.

S270, migrating the file items to be migrated corresponding to the scanning information in the scanning list to a target file storage system, wherein the scanning information comprises: inventory scan information and incremental scan information.

According to the technical scheme of the embodiment of the invention, the stock scanning information is obtained by acquiring the task to be migrated and scanning each file item in the task to be migrated, and the stock scanning information is written into the scanning list; acquiring a target file item operation executed by a task to be migrated after scanning starts, determining incremental scanning information according to the target file item operation, and writing the incremental scanning information into a scanning list; migrating the file items to be migrated corresponding to the scanning information in the scanning list to a target file storage system, wherein the scanning information comprises: the storage scanning information and the increment scanning information solve the problems that in the prior art, when a file item is operated in the scanning process and is moved, the scanning item cannot be found in the original storage and the scanning item needs to be scanned repeatedly and compared completely, increment changes of the file data in the scanning process can be obtained and recorded in real time, a scanning list is modified, repeated scanning is not needed, the consistency of file data movement can be guaranteed, the movement complexity is reduced, and the movement efficiency is improved.

On the basis of the above embodiment, step S240: scanning the attribute information of the current stock file item contained in the current-level directory according to the judgment result to obtain the stock scanning information of the current-level directory, wherein the method comprises the following steps of:

if the judgment result is that the path of the current-level directory exists in the target file storage system, determining stock scanning information of the current-level directory according to the attribute information of the current stock file item meeting a first preset condition;

and if the judgment result is that the path of the current-level directory does not exist in the target file storage system, determining stock scanning information of the current-level directory according to the attribute information of the current-level directory.

Specifically, if the judgment result is that the path of the previous-stage directory exists in the target file storage system, the current-stage directory already exists in the target file storage system, the current-stage directory does not need to be migrated, and the stock scanning information only needs to include the scanning information of the current stock file item under the current-stage directory; however, a file item which conflicts with the current stock file item may exist in the target storage system, so that the current stock file item cannot be migrated into the target storage system, and therefore, the current stock file item still needs to meet a first preset condition, and the scanning information of the current stock file item can be determined as the stock scanning information of the current-level directory.

If the judgment result is that the current-level directory does not exist in the target file storage system, the file items contained in the current-level directory need to be migrated, and the inventory scanning information needs to include the scanning information of the current file items in the current-level directory. Therefore, the stock scan information of the current-level directory is determined based on the attribute information of the current-level file item contained in the current-level directory.

It can be understood that the manner of determining the scanning information of the current stock file item according to the attribute information of the current stock file item (including the current stock file or the next-level directory) is the same as the manner of "determining the stock scanning information of the first-level directory according to the attribute information of the first-level stock file item" in this embodiment, and details of this embodiment of the present invention are not repeated.

Optionally, the types of the current stock file items may include: a directory or a file.

If the type of the current stock file item is a directory, the first preset condition comprises: the name of a first target directory corresponding to the path of the current stock file item in the target file storage system is different from the name of the current stock file item;

if the type of the current stock file item is a file, the first preset condition comprises that: and the name of the target file corresponding to the path of the current stock file item in the target file storage system is different from the name of the current stock file item, or the updating time of the current stock file item is later than that of the target file.

Specifically, if the type of the current stock file item is a directory, if a target directory exists in the target file storage system, and the name and the path of the target directory are the same as those of the current stock file item, a conflict occurs. Therefore, when the type of the current stock file item is a directory, the first preset condition to be met includes: the name of the first target directory corresponding to the path of the current stock file item in the target file storage system is different from the name of the current stock file item.

If the file of the current stock file item is a file, a target file with the name and the path identical to those of the current stock file item exists in the target file storage system, and the update time of the current stock file item is earlier than or equal to that of the target file, a conflict occurs. Therefore, when the type of the current stock file item is a file, the first preset condition to be met includes: the name of the target file corresponding to the path of the current stock file item in the target file storage system is different from the name of the current stock file item, or the update time of the current stock file item is later than that of the target file.

On the basis of the above embodiment, step S210: respectively writing the index numbers of all levels of directories of the task to be migrated into the storage to-be-scanned queue, and reading all current levels of directories in the storage to-be-scanned queue through the scanning thread, wherein the method comprises the following steps:

s211, writing the index number of the primary catalog of the task to be migrated into the inventory to-be-scanned queue, and reading the index number of the catalog written into the inventory to-be-scanned queue through a scanning thread.

Specifically, the index number of the first-level directory of the task to be migrated is written into the queue to be scanned, the initial state of the queue to be scanned may be an empty queue, and the index number of the directory written in the queue to be scanned is read through the scanning thread.

S212, determining each next-level stock file item of the inventory in the queue to be scanned, which is read by the scanning thread.

After the queue to be scanned for inventory finishes reading, the queue to be scanned for inventory is restored to an empty queue, and a next-level inventory file item of the inventory in the queue to be scanned for inventory read by the scanning thread is determined (if the inventory in the queue to be scanned for inventory read by the scanning thread is a first-level inventory, the next-level inventory file item is a second-level inventory file item).

And S213, for each next-level stock file item, if the type of the next-level stock file item is a directory, writing the index number of the next-level stock file item into the stock to-be-scanned queue.

Specifically, for each next-level inventory file item, the type of the next-level inventory file item is determined, if the type of the next-level inventory file item is a directory (if a scanning thread reads the directory in the inventory to-be-scanned queue as the first-level directory, the next-level inventory file item of the type as the directory is the second-level directory), the next-level inventory file item of the type as the directory may further include a next-level inventory file item (if the next-level inventory file item is the second-level directory, the next-level inventory file item is the third-level inventory file item), and therefore, the index number of the next-level inventory file item of the type as the directory needs to be written into the inventory to-be-scanned queue, so that the next-level directory is scanned according to the inventory to-be-scanned queue to obtain the next-level inventory file item of the next-level included in the next-level directory.

If the type of the next-level stock file item is a file (if the directory in the queue to be scanned is read by the scanning thread as the first-level directory, the next-level stock file item with the type of the file is a second-level file), the next-level stock file item with the type of the file does not contain any file item any more, so that the index number of the next-level stock file item with the type of the file does not need to be written into the queue to be scanned.

And S214, reading the index number of the directory written in the inventory queue to be scanned through the scanning thread.

Specifically, after the writing of the index number of the directory is completed in the queue to be scanned, the index number of the directory written in the queue to be scanned is read through the scanning thread.

And S215, returning to execute the determination operation of each next-level stock file item until the traversal of each level of stock file item contained in the task to be migrated is completed.

Specifically, after the reading of the stock to-be-scanned queue is completed, the stock to-be-scanned queue restores the initial state of the empty queue again, and the steps S212 to S215 are executed again until the traversal of the stock file items at each level included in the to-be-migrated task is completed, so that the stock scanning process of the file items at each level in the to-be-migrated task is completed.

According to the technical scheme of the embodiment, all levels of directories in the task to be migrated can be obtained by sequentially traversing all levels of file items in the task to be migrated from top to bottom, so that all levels of directories in the task to be migrated are scanned to obtain stock scanning information, the scanning information is ensured not to be omitted, and the task of reading all levels of directories in the task to be migrated is executed concurrently through the scanning thread, so that the scanning efficiency can be improved.

EXAMPLE III

Fig. 3 is a flowchart of a file data migration method according to a third embodiment of the present invention, and in this embodiment, based on the above embodiment, further details the technical feature "determining incremental scanning information according to a target file item operation" in step S120, where the target file item operation includes: file item creation operations and file item renaming operations, the file item renaming operations comprising: directory rename operations and file rename operations. As shown in fig. 3, the method includes:

s310, acquiring the task to be migrated, scanning each file item in the task to be migrated to obtain stock scanning information, and writing the stock scanning information into a scanning list.

And S320, acquiring the target file item operation executed by the task to be migrated after the scanning starts.

S330, if the operation of the target file item is a directory renaming operation and a target directory corresponding to the directory renaming operation does not exist in the scanning list, determining the renamed directory as an incremental directory, and writing the index number of the incremental directory into an incremental to-be-scanned list; and respectively scanning each increment directory in the increment list to be scanned to obtain increment scanning information of the increment directory.

The incremental to-be-scanned list is used for writing an incremental file entry to be scanned, and the incremental file entry may include an incremental directory.

Specifically, if the target file item operation is a directory renaming operation and a target directory corresponding to the directory renaming operation does not exist in the scan list, that is, the directory is renamed, and the renamed directory is determined to be an incremental directory; and further judging whether the target directory subjected to the directory renaming operation exists in the scan list or not, if not, indicating that the target directory is not scanned, and writing the target directory into the scan list in the subsequent scanning process. However, the scanning process and the directory renaming operation process may occur simultaneously, so that the target directory is not scanned before and after renaming, and therefore, if the target directory corresponding to the directory renaming operation does not exist in the scan list, the index number of the incremental directory still needs to be written into the incremental to-be-scanned list, thereby avoiding missing scanning. And for the increment directory with the index number written in the increment scanning list, performing increment scanning to obtain the attribute information of the increment directory corresponding to the index number, and determining the increment scanning information of the increment directory according to the attribute information of the increment directory. If the rename directory is already written into the scan list, the incremental scan information may be written into the scan list over the original scan information.

If the type of the renamed file item is a file, only the file data in the file does not include any file item, and the file does not need to be further scanned. Therefore, the index number of the file to which the renaming operation is performed does not need to be added to the scan list or the incremental to-be-scanned list.

It should be noted that, because the primary directory storing the task to be migrated in the original file storage system does not need to be migrated to the target file storage system, that is, the primary directory storing the task to be migrated in the target file storage system is not limited by the directory renaming operation, the incremental directory determined according to the directory renaming operation does not include the primary directory.

S340, if the operation of the target file item is a file renaming operation and the target file corresponding to the file renaming operation does not exist in the scanning list, determining incremental scanning information according to the attribute information of the target file.

Specifically, if the operation of the target file item is a file renaming operation and the target file corresponding to the file renaming operation does not exist in the scan list, it indicates that the target file is not scanned and can be written into the scan list in a subsequent scanning process. However, the scanning process and the file renaming operation process may occur simultaneously, so that the target file is not scanned before and after renaming, and therefore, if the target file corresponding to the file renaming operation does not exist in the scan list, the incremental scan information is determined according to the attribute information of the target file. If the renamed file is already written into the scan list, the incremental scan information can be written into the scan list to overwrite the original scan information.

And S350, if the target file item operation is a file item creation operation, determining incremental scanning information according to the attribute information of the created file item.

Specifically, if the target file item operation is a file item creation operation, that is, a file or a directory is created in the task to be migrated after the scanning starts, the scan list does not include the created file item, and therefore, the incremental scan information needs to be determined according to the attribute information of the created file item and the operation information of the creation operation.

Exemplary attribute information of the created file item may also include: the index number, name of the created file entry, and the index number of the parent directory of the created file entry. Of course, it may also include: creation time, creation type, and the like.

And S360, writing the incremental scanning information into the scanning list.

Specifically, in steps S330 to S350, the incremental scanning information of the file item determined according to the file item renaming operation and/or the file item creating operation is written into the scanning list.

S370, migrating the file items to be migrated corresponding to the scanning information in the scanning list to a target file storage system, wherein the scanning information comprises: inventory scan information and incremental scan information.

According to the technical scheme of the embodiment of the invention, the stock scanning information is obtained by acquiring the task to be migrated and scanning each file item in the task to be migrated, and the stock scanning information is written into the scanning list; acquiring a target file item operation executed by a task to be migrated after scanning starts, if the target file item operation is a directory renaming operation, determining a renamed directory as an incremental directory, and writing an index number of the incremental directory into an incremental list to be scanned; respectively scanning each increment catalog in the increment list to be scanned to obtain increment scanning information of the increment catalog; if the target file item operation is a file item creation operation, determining incremental scanning information according to attribute information of the created file item; writing the incremental scan information into a scan list; migrating the file items to be migrated corresponding to the scanning information in the scanning list to a target file storage system, wherein the scanning information comprises: the method and the device have the advantages that the problems that in the prior art, scanning items cannot be found in original storage and repeated full scanning comparison is needed when the file items are moved due to operation in the scanning process are solved, increment changes of the file data in the scanning process can be obtained and recorded in real time and are synchronized into a scanning list, repeated scanning is not needed, the consistency of file data movement can be guaranteed, movement complexity is reduced, and movement efficiency is improved.

On the basis of the foregoing embodiment, the step of scanning each increment directory in the increment to-be-scanned list in step S330 to obtain increment scanning information of the increment directory may specifically include:

step S331, writing the index number of each incremental directory into an incremental queue to be scanned, and reading each incremental directory written in the incremental queue to be scanned through a scanning thread.

The queue to be scanned is a queue for writing index numbers of the incremental catalog generated in the incremental scanning process. The scanning threads can execute tasks for reading the increment directories in the increment to-be-scanned queue in parallel, and the maximum number of the tasks executed by each scanning thread can be configured according to actual requirements and hardware resource states. Because the incremental scanning process needs to be executed after the stock scanning process is completed, the stock scanning process of the incremental scanning process can be completed by adopting one scanning thread.

Step S332, for each incremental directory, scanning the incremental directory to obtain attribute information of each current incremental file item included in the incremental directory.

The current incremental file items refer to file items contained in an incremental directory newly added by a task to be migrated in the original file storage system after scanning starts. The current delta file entry includes the current delta file under the delta directory and/or a next level directory of the delta directory. For example, if the delta directory is a primary directory, the file entries contained in the primary directory are primary delta file entries.

Specifically, the incremental directory is scanned, so that all current incremental file items included in the incremental directory can be obtained, and the attribute information of each current incremental file item is obtained.

Illustratively, the attribute information of the current delta file item at least includes: the index number of the file entry, the name of the file entry, and the parent directory index number of the file entry (i.e., the index number of the delta directory).

Step S333, for each current incremental file item, determining incremental scanning information of the incremental directory according to the attribute information of the current incremental file item.

Specifically, the implementation manner of determining the incremental scanning information of the incremental directory according to the attribute information of the current incremental file item is substantially the same as the implementation process of determining the stock scanning information of the current-level directory according to the attribute information of the current stock file item in the above embodiment, and details of this implementation are not repeated again in the embodiments of the present invention.

Example four

Fig. 4 is a flowchart of a file data migration method according to a fourth embodiment of the present invention, where this embodiment further details technical features "migrating to-be-migrated file items respectively corresponding to each piece of scanning information in the scan list to a target file storage system" in step S120 on the basis of any one of the above embodiments. As shown in fig. 4, the method includes:

s410, acquiring the task to be migrated, scanning each file item in the task to be migrated to obtain stock scanning information, and writing the stock scanning information into a scanning list.

S420, acquiring a target file item operation executed by the task to be migrated after the scanning starts, determining incremental scanning information according to the target file item operation, and writing the incremental scanning information into a scanning list.

S430, reading first preset number of pieces of target to-be-migrated scanning information from each piece of scanning information in the scanning list.

The scanning information in the scanning list may include scanning information of file items to be migrated and file items that have been migrated, the target scanning information to be migrated is scanning information of file items to be migrated read from the scanning information in the scanning list, the number of target scanning information to be migrated per reading may be a first preset number, and the first preset number may be configured according to actual requirements and use conditions of hardware resources.

Specifically, the scanning information of the file items to be migrated is determined from the scanning information of the scanning list, and the first preset number of target scanning information to be migrated is read from the scanning information of the file items to be migrated. In order to facilitate the migration of the file items to the target file storage system for storage, the first preset number of pieces of target to-be-migrated scanning information may be read sequentially according to a writing order of the scanning information in the scanning list.

It should be noted that each piece of scanning information in the scan list may include scanning information of a file item to be migrated and a file item that has been migrated, and a flag bit may be set to distinguish a migration state of the file item, for example, a flag bit of a file item that has not been migrated is 1, a flag bit of a file item that has been migrated is 0, and scanning information of a file item whose flag bit is 1 is obtained, that is, scanning information of the file item to be migrated may be obtained.

And S440, according to the target to-be-migrated scanning information, migrating the target to-be-migrated file items corresponding to the target to-be-migrated scanning information to a target file storage system in a preset migration mode.

Specifically, the file item corresponding to the target to-be-migrated scanning information may be determined as the target to-be-migrated file item, the type and the target storage path corresponding to the target to-be-migrated file item may be determined according to the target to-be-migrated scanning information, and the target to-be-migrated file item may be migrated to the target file storage system in a preset migration manner.

For example, the manner of determining the target storage path of the target file item to be migrated may be to obtain a target storage location of the target file storage system, that is, a location of a first directory in the target file storage system, where the task to be migrated is stored, determine an original storage path of the target file item to be migrated in the original file storage system according to the target scanning information to be migrated, and determine the target storage path of the target file item to be migrated according to the original storage path and the target storage location.

For example, the type corresponding to the target file item to be migrated may be a file or a directory. The files can be migrated in batches, and the directories can only be migrated in sequence, so that the target file items to be migrated are migrated to the target file storage system in a preset migration mode according to the types of the target file items to be migrated and corresponding to the target scanning information to be migrated and the first target storage path.

And S450, returning to execute the acquisition operation of the target scanning information to be migrated until all the file items to be migrated corresponding to the scanning information in the scanning list are migrated to the target file storage system.

Specifically, after the target to-be-migrated file items corresponding to the first number of pieces of target to-be-migrated scanning information are migrated, the step S430 of obtaining the target to-be-migrated scanning information is performed, and the first preset number of pieces of target to-be-migrated scanning information are read from each piece of scanning information in the scanning list until all the to-be-migrated file items corresponding to all the pieces of scanning information in the scanning list are migrated to the target file storage system.

According to the technical scheme of the embodiment of the invention, the stock scanning information is obtained by acquiring the task to be migrated and scanning each file item in the task to be migrated, and the stock scanning information is written into the scanning list; acquiring a target file item operation executed by a task to be migrated after scanning starts, determining incremental scanning information according to the target file item operation, and writing the incremental scanning information into a scanning list; reading a first preset number of pieces of target to-be-migrated scanning information from each piece of scanning information in a scanning list; for target file items to be migrated corresponding to the target scanning information to be migrated, respectively, determining a first initial storage path and a first target storage position of the target file items to be migrated according to the target scanning information to be migrated, and determining a first target storage path of the target file items to be migrated according to the first initial storage path and the first target storage position; migrating each target file item to be migrated to a target file storage system in a preset migration mode according to the type and the first target storage path corresponding to each target file item to be migrated respectively; the method comprises the steps of returning to execute the acquisition operation of target scanning information to be migrated until all the file items to be migrated corresponding to all the scanning information in a scanning list are migrated to a target file storage system, solving the problems that in the prior art, when the file items are migrated due to target operation in the scanning process, the scanning items cannot be found in the original storage, and repeated full scanning comparison is needed, being capable of acquiring and recording the increment change of file data in the scanning process in real time, modifying the scanning list, not needing repeated scanning, ensuring the consistency of file data migration, reducing migration complexity and improving migration efficiency.

On the basis of the above embodiment, step S450: migrating each target file item to be migrated corresponding to each target scanning information to a target file storage system in a preset migration mode, wherein the migration mode comprises the following steps:

s451, adding target to-be-migrated scanning information corresponding to each target to-be-migrated file item with the type being a file into a stock to-be-migrated queue; and migrating the target file items to be migrated corresponding to the target scanning information to be migrated in the stock file to be migrated queue to the target file storage system in parallel through a migration thread.

Specifically, for each target file item to be migrated with the type of a file, the target scanning information to be migrated corresponding to the target file item to be migrated may be migrated in a parallel migration manner, so that the target scanning information to be migrated corresponding to each target file item to be migrated with the type of a file in the first preset number of target file items to be migrated is added to the stock file to be migrated queue, and the target file item to be migrated corresponding to each target scanning information to be migrated in the stock file to be migrated queue is migrated in parallel to the target file storage system through the migration thread.

And S452, serially migrating each target file item to be migrated with the type of the directory to the target file storage system.

Specifically, for the target files to be migrated which are of the directory type and cannot be migrated in parallel, the files are sequentially migrated to the target file storage system in a serial mode.

According to the embodiment of the invention, the file items to be migrated are migrated to the target file storage system in a parallel or serial mode according to the type of the file items to be migrated, so that the migration rate can be improved.

EXAMPLE five

Fig. 5 is a flowchart of a file data migration method according to a fifth embodiment of the present invention, where in this embodiment, on the basis of the first embodiment, the target file item operation includes: file item modification operation, file item deletion operation and file renaming operation; the method further includes, after acquiring the target file item operation executed on the task to be migrated after the scanning is started in step S120:

if the target file item operation is a file item modification operation, modifying the attribute information of the file items in the scanning list according to the modified file items;

if the target file item operation is a file item deleting operation and the file item is in the scanning list, judging whether the file item is migrated or not; if not, deleting the attribute information of the file item from the scanning list; if yes, deleting the attribute information of the file item from the scanning list and writing the operation information of the deletion operation into an increment operation list of the target file storage system;

if the target file item operation is a file renaming operation and the file item is in the scanning list, judging whether the file item is migrated or not; if not, modifying the attribute information of the file items in the scanning list according to the renaming operation; and if so, writing the operation information of the renaming operation into an increment operation list of the target file storage system.

As shown in fig. 5, the method includes:

and S510, acquiring a task to be migrated, scanning each file item in the task to be migrated to obtain stock scanning information, and writing the stock scanning information into a scanning list.

S520, acquiring a target file item operation executed by the task to be migrated after the scanning starts; the target file item operation includes: file item modification operations, file item deletion operations, and file renaming operations.

S530, if the target file item operation is a file item modification operation, modifying the attribute information of the file items in the scanning list according to the modified file items.

The file item modification operation refers to modifying the content or file item format and the like contained in the file item, and does not include modifying the name of the corresponding file item; the file item modifying operation includes a file modifying operation and a directory modifying operation.

Specifically, if the file item modification operation executed on the file in the task to be migrated is acquired after the scanning starts, the scanning information of the corresponding file item to which the file item modification operation is executed in the scanning list is directly modified according to the modified file item.

It should be noted that the file item modification operation performed on the file item does not change the scan information such as the storage path of the file item, and the scan information that may be changed may be, for example, information such as the file size, the file format, the file content, or the directory content.

S540, if the target file item operation is a file item deleting operation and the scanning information of the deleted file item is in the scanning list, judging whether the file item is migrated or not; if not, deleting the scanning information of the deleted file item from the scanning list; if yes, deleting the attribute information of the deleted file item from the scanning list and writing the operation information of the deletion operation into an increment operation list of the target file storage system.

The file item deleting operation comprises a file deleting operation and a directory deleting operation. The operation information of the deletion operation includes information such as scan information of the deleted file item and operation identification of the operation of deleting the file item.

Specifically, if the file item deletion operation executed on the file item in the task to be migrated is acquired after the scanning starts, and the deleted file item is not in the scan list, it indicates that the scanning information of the deleted file item is not scanned in the scan list yet, and no operation needs to be executed; if the deleted file item is in the scan list, it indicates that the deleted file item has been scanned, and it needs to be determined whether the deleted file item has been migrated; if not, deleting the scanning information of the deleted file item from the scanning list, and avoiding the error that the file cannot be found when data migration is carried out according to the scanning information in the scanning list. If the deleted file item is in the scanning list; if the file item is migrated, the deleted file item still exists in the target file storage system, and therefore, it is necessary to delete the scan information of the deleted file item from the scan list and write the operation information of the deletion operation into an incremental operation list of the target file storage system, where the incremental operation list is used to instruct the target file storage system to perform a corresponding operation on the stored file item.

For example, the manner of writing the operation information of the delete operation into the increment operation list of the target file storage system is to write the operation information of the delete operation into the increment operation list in the form of an operation log, which is not limited in this embodiment of the present invention.

S550, if the operation of the target file item is a file renaming operation and the scanning information of the renamed file is in the scanning list, judging whether the renamed file is migrated or not; if not, modifying the scanning information of the renamed file in the scanning list; and if so, modifying the scanning information of the renamed file in the scanning list, and writing the operation information of the rename operation into an increment operation list of the target file storage system.

The operation information of the file renaming operation comprises the scanning information of the renamed file item, the operation identification of the renaming file operation and other information.

Specifically, if the file renaming operation executed on the file item in the task to be migrated is acquired after the scanning starts, and the renamed file is in the scanning list, it is indicated that the renaming file has been scanned, and it is necessary to determine whether the renamed file has been migrated; if not, modifying the scanning information of the renamed file in the scanning list, namely modifying the name in the scanning information corresponding to the file item which is subjected to the renaming operation in the scanning list into the renamed name, and avoiding the error that the file cannot be found when data migration is carried out according to the scanning information in the scanning list. If the renamed file item is in the scanning list; if the file is migrated, the name of the file which is stored in the target file storage system and migrated in the target file storage system is still the name before renaming the file, so that the scanning information of the renamed file in the scanning list needs to be modified to avoid the condition that the scanning information in the scanning list is inconsistent with the information of the migrated file, and the operation information of the renaming operation is written into an increment operation list of the target file storage system, wherein the increment operation list is used for indicating the target file storage system to execute corresponding operation on the stored file item.

And S560, acquiring a second preset number of target to-be-synchronized operation information from the operation information of the incremental operation list according to the latest operation time sequence of the file items.

The operation information in the incremental operation list may include operation information of the file items to be synchronized and the synchronized file items, the target operation information to be synchronized is operation information of the file items to be synchronized read from the operation information in the incremental operation list, the number of the target operation information to be synchronized each time is read may be a second preset number, and the second preset number may be configured according to actual requirements and use conditions of hardware resources.

Specifically, the operation information of the file items to be synchronized is determined from the operation information of the incremental operation list, and a second preset number of target operation information to be synchronized is read from the operation information of the file items to be synchronized. In order to facilitate the synchronization of the file items to the target file storage system, the second preset number of target to-be-synchronized operation information may be read sequentially according to the latest operation time sequence of the file items.

It should be noted that each operation information in the operation list may include operation information of a file item to be synchronized and a synchronized file item, and a flag bit may be set to distinguish a synchronization state of the file item, for example, a flag bit of an unsynchronized file item is 1, a flag bit of a synchronized file item is 0, and operation information of a file item with a flag bit of 1 is obtained, i.e., operation information of the file item to be synchronized can be obtained.

And S570, executing synchronous operation on the target file items to be synchronized corresponding to the target information to be synchronized in the target file storage system according to the information of the target information to be synchronized.

Specifically, a target storage path and an index number of a target file item to be synchronized are determined according to the target file item to be synchronized operation information, so that the target file item to be synchronized in the target file storage system is positioned, and the synchronization operation is performed on the target file item to be synchronized.

For example, the manner of determining the target storage path of the target file item to be migrated may be to obtain a target storage location of the target file storage system, that is, a location of the first directory in the target file storage system, where the task to be migrated is stored, determine an original storage path of the target file item to be migrated in the original file storage system according to the target scanning information to be migrated, and determine the target storage path of the target file item to be migrated according to the original storage path and the target storage location.

And S580, returning to execute the acquisition operation of the target to-be-synchronized scanning information until the to-be-synchronized file items corresponding to all the operation information in the incremental operation list execute the synchronization operation.

Specifically, after the target to-be-synchronized file items corresponding to the second amount of target to-be-synchronized operation information complete the synchronization operation, the step S560 of obtaining the target to-be-synchronized operation information is executed, and the second preset amount of target to-be-synchronized operation information is read from the operation information in the incremental operation list until all the to-be-synchronized file items corresponding to the operation information in the incremental operation list execute the synchronization operation.

According to the technical scheme of the embodiment of the invention, the stock scanning information is obtained by acquiring the task to be migrated and scanning each file item in the task to be migrated, and the stock scanning information is written into the scanning list; acquiring a target file item operation executed by a task to be migrated after scanning starts; the target file item operation includes: file item modification operation, file item deletion operation and file renaming operation; if the target file item operation is a file item modification operation, modifying the attribute information of the file items in the scanning list according to the modified file items; if the target file item operation is a file item deleting operation and the scanning information of the deleted file item is in the scanning list, judging whether the file item is migrated or not; if not, deleting the scanning information of the deleted file item from the scanning list; if yes, deleting the attribute information of the deleted file item from the scanning list and writing the operation information of the deletion operation into an increment operation list of the target file storage system; if the operation of the target file item is a file renaming operation and the scanning information of the renamed file is in the scanning list, judging whether the renamed file is migrated or not; if not, modifying the scanning information of the renamed file in the scanning list; if so, modifying the scanning information of the renamed file in the scanning list, and writing the operation information of the renamed operation into an increment operation list of the target file storage system; acquiring a second preset number of target to-be-synchronized operation information from the operation information of the incremental operation list according to the latest operation time sequence of the file items; according to the information of the operation to be synchronized of each target, carrying out synchronization operation on the file items to be synchronized of the target corresponding to the information of the operation to be synchronized of each target in the target file storage system; and returning to execute the acquisition operation of the target information to be synchronously scanned until all the items of the files to be synchronized corresponding to all the operation information in the incremental operation list execute the synchronous operation, so that the incremental change of the file data in the scanning process can be acquired and recorded in real time, the consistency of file data migration can be ensured without repeated scanning, the migration complexity is reduced, and the migration efficiency is improved.

Example six

Fig. 6 is a schematic structural diagram of a file data migration apparatus according to a sixth embodiment of the present invention. As shown in fig. 6, the apparatus includes: an inventory scan module 610, an incremental scan module 620, and a migration module 630;

the stock scanning module 610 is configured to obtain a task to be migrated, scan each file item in the task to be migrated to obtain stock scanning information, and write the stock scanning information into a scanning list;

an incremental scanning module 620, configured to acquire a target file item operation performed on the task to be migrated after scanning starts, determine incremental scanning information according to the target file item operation, and write the incremental scanning information into a scanning list;

a migration module 630, configured to migrate the file items to be migrated corresponding to each piece of scanning information in the scanning list to a target file storage system, where the scanning information includes: the inventory scan information and the incremental scan information.

Optionally, the inventory scan module 610 includes:

the first stock reading unit is used for respectively writing the index numbers of all levels of catalogues of the task to be migrated into a stock to-be-scanned queue and reading all current levels of catalogues in the stock to-be-scanned queue through a scanning thread;

a current-level directory scanning unit, configured to scan, for each current-level directory, the current-level directory to obtain attribute information of a current stock file item included in the current-level directory;

a first stock scanning information determining unit, configured to determine stock scanning information of the primary directory according to attribute information of a primary stock file item included in the primary directory if the current-level directory is the primary directory;

and a second stock scan information determining unit, configured to determine whether a path of the current-level directory exists in the target file storage system if the current-level directory is not a first-level directory, and determine stock scan information of the current-level directory according to a determination result and the attribute information of the current stock file item.

Optionally, the second stock scan information determining unit is specifically configured to:

and if the judgment result is that the path of the current-level directory does not exist in the target file storage system, determining stock scanning information of the current-level directory according to the attribute information of the current stock file item contained in the current-level directory.

Optionally, if the type of the current stock file item is a directory, the first preset condition includes: the name of a first target directory corresponding to the path of the current stock file item in the target file storage system is different from the name of the current stock file item;

Optionally, the first inventory reading unit includes:

writing the index number of the primary catalog of the task to be migrated into the inventory to-be-scanned queue, and reading the index number of the catalog written into the inventory to-be-scanned queue through a scanning thread;

determining each next-level stock file item of the directory in the stock to-be-scanned queue read by the scanning thread;

for each next-level stock file item, if the type of the next-level stock file item is a directory, writing the index number of the next-level stock file item into the stock to-be-scanned queue;

reading the index number of the directory written in the inventory queue to be scanned through the scanning thread;

and returning to execute the determination operation of each next-level stock file item until the traversal of each level of stock file item contained in the task to be migrated is completed.

Optionally, the target file item operation includes: file item creation operations and file item renaming operations, the file item renaming operations comprising: directory renaming operations and file renaming operations, the incremental scanning module 620 includes:

a first incremental scanning information determining unit, configured to determine, if the target file entry operation is a directory renaming operation and a target directory corresponding to the target file entry operation does not exist in the scanning list, a renamed directory as an incremental directory, and write an index number of the incremental directory into an incremental to-be-scanned list; respectively scanning each increment catalogue in the increment list to be scanned to obtain increment scanning information of the increment catalogue;

a second incremental scanning information determining unit, configured to determine incremental scanning information according to attribute information of the target file if the target file item operation is a file renaming operation and the target file corresponding to the file renaming operation does not exist in the scanning list;

and a third incremental scanning information determining unit, configured to determine incremental scanning information according to attribute information of the created file item, if the target file item operation is a file item creation operation.

Optionally, the first incremental scanning information determining unit is specifically configured to:

respectively writing the index number of each increment directory into an increment queue to be scanned, and reading each increment directory written in the increment queue to be scanned through a scanning thread;

for each increment directory, scanning the increment directory to obtain attribute information of each current increment file item contained in the increment directory;

and for each current incremental file item, determining incremental scanning information of each incremental directory according to the attribute information of each current incremental file item.

Optionally, the migration module 630 includes:

the reading unit is used for reading a first preset number of pieces of target to-be-migrated scanning information from each piece of scanning information of the scanning list;

a storage path determining unit, configured to migrate, according to each piece of target to-be-migrated scanning information, each target to-be-migrated file item corresponding to each piece of target to-be-migrated scanning information to a target file storage system in a preset migration manner;

and the return execution unit is used for returning and executing the acquisition operation of the target scanning information to be migrated until all the file items to be migrated corresponding to the scanning information in the scanning list are migrated to the target file storage system.

Optionally, the migration unit is specifically configured to:

adding target to-be-migrated scanning information corresponding to each target to-be-migrated file item with the type of a file into a stock to-be-migrated queue; migrating target file items to be migrated corresponding to the target scanning information to be migrated in the stock file to be migrated queue to the target file storage system in parallel through a migration thread;

and serially migrating each target file item to be migrated with the type of the directory to the target file storage system.

Optionally, the target file item operation includes: the file data migration method comprises the following steps of file item modification operation, file item deletion operation and file renaming operation, and correspondingly, the file data migration further comprises the following steps:

a first modification module, configured to modify, if the target file item operation is a file item modification operation, scan information of the file item in the scan list according to the modified file item;

a deleting module, configured to determine whether the file item has been migrated if the target file item operation is a file item deleting operation and the scanning information of the deleted file item is in the scanning list; if not, deleting the scanning information of the deleted file item from the scanning list; if so, deleting the scanning information of the deleted file item from the scanning list, and writing the operation information of the file item deleting operation into an increment operation list of the target file storage system;

the second modification module is used for judging whether the renamed file is migrated or not if the target file item operation is a file renaming operation and the scanning information of the renamed file is in the scanning list; if not, modifying the scanning information of the renamed file in the scanning list; if so, modifying the scanning information of the renamed file in the scanning list, and writing the operation information of the file renaming operation into an increment operation list of the target file storage system;

wherein the operation information comprises scanning information of a file item on which the target file item operation is performed and an identification of the target file item operation.

Optionally, the method further includes:

the reading module is used for reading a second preset number of pieces of target to-be-synchronized operation information from the operation information in the incremental operation list according to the latest operation time sequence of the file items;

the synchronization module is used for executing synchronization operation on the target file items to be synchronized corresponding to the target file information to be synchronized in the target file storage system according to the target information to be synchronized;

and the return execution module is used for returning and executing the acquisition operation of the target information to be synchronized and scanned until all the file items to be synchronized corresponding to the operation information in the increment operation list execute the synchronization operation.

Optionally, the attribute information of the file item at least includes: the index number of the file entry, the name of the file entry, the parent directory index number of the file entry, and the type of the file entry.

Optionally, the target file storage system includes: an unstructured data management system.

The file data migration device provided by the embodiment of the invention can execute the file data migration method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

EXAMPLE seven

FIG. 7 shows a schematic block diagram of a computer device 10 that may be used to implement an embodiment of the invention. Computer devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The computer device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.

As shown in fig. 7, the computer device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM)12, a Random Access Memory (RAM)13, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 11 can perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM)12 or the computer program loaded from a storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data necessary for the operation of the computer device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14. The storage unit 18 is also provided with an original file storage system.

A number of components in the computer device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, or the like; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the computer device 10 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.

The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. The processor 11 performs the various methods and processes described above, such as a file data migration method.

In some embodiments, the file data migration method may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the computer device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the file data migration method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the file data migration method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

A computer program for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), blockchain networks, and the internet.

The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solution of the present invention can be achieved.

The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A file data migration method is applied to an original file storage system, and comprises the following steps:

migrating the file items to be migrated corresponding to the scanning information in the scanning list to a target file storage system, wherein the scanning information comprises: the stock scan information and the incremental scan information.

2. The method according to claim 1, wherein the scanning each file item in the task to be migrated to obtain inventory scanning information comprises:

respectively writing the index numbers of all levels of directories of the task to be migrated into a storage to-be-scanned queue, and reading all current levels of directories in the storage to-be-scanned queue through a scanning thread;

scanning the current-level directory to obtain attribute information of a current stock file item contained in the current-level directory aiming at each current-level directory;

if the current-level directory is a first-level directory, determining stock scanning information of the first-level directory according to attribute information of a first-level stock file item contained in the first-level directory;

if the current-level directory is not a first-level directory, judging whether the path of the current-level directory exists in the target file storage system, and determining stock scanning information of the current-level directory according to the judgment result and the attribute information of the current stock file item.

3. The method of claim 2, wherein said determining inventory scan information for said current level directory based on the determination and attribute information for said current inventory file item comprises:

4. The method of claim 3, wherein if the type of the current stock file item is a directory, the first preset condition comprises: the name of a first target directory corresponding to the path of the current stock file item in the target file storage system is different from the name of the current stock file item;

5. The method according to claim 2, wherein the writing the index numbers of the directories at each level of the task to be migrated into the queue to be scanned at the inventory level respectively, and reading the directories at each current level in the queue to be scanned at the inventory level through a scanning thread comprises:

6. The method of claim 1, wherein the target file item operation comprises: file item creation operations and file item renaming operations, the file item renaming operations comprising: determining incremental scanning information according to the target file item operation, wherein the directory renaming operation and the file renaming operation comprise:

if the target file item operation is a directory renaming operation and a target directory corresponding to the directory renaming operation does not exist in the scanning list, determining the renamed directory as an incremental directory, and writing the index number of the incremental directory into an incremental to-be-scanned list; respectively scanning each increment catalogue in the increment list to be scanned to obtain increment scanning information of the increment catalogue;

if the target file item operation is a file renaming operation and a target file corresponding to the file renaming operation does not exist in the scanning list, determining incremental scanning information according to attribute information of the target file;

and if the target file item operation is a file item creation operation, determining incremental scanning information according to the attribute information of the created file item.

7. The method of claim 6, wherein the scanning each incremental directory in the incremental to-be-scanned list to obtain incremental scanning information of the incremental directory comprises:

8. The method according to any one of claims 1 to 7, wherein the migrating the file items to be migrated corresponding to the respective scanning information in the scanning list to a target file storage system includes:

reading a first preset number of pieces of target to-be-migrated scanning information from each piece of scanning information in the scanning list;

according to the target to-be-migrated scanning information, migrating each target to-be-migrated file item corresponding to the target to-be-migrated scanning information to a target file storage system in a preset migration mode;

and returning to execute the acquisition operation of the target scanning information to be migrated until all the file items to be migrated corresponding to the scanning information in the scanning list are migrated to the target file storage system.

9. The method according to claim 8, wherein the migrating each target file item to be migrated corresponding to each target scanning information to a target file storage system in a preset migration manner includes:

10. The method of claim 6, wherein the target file item operation further comprises: correspondingly, after obtaining the target file item operation executed on the task to be migrated after the scanning starts, the method further includes:

if the target file item operation is a file item modification operation, modifying the scanning information of the file items in the scanning list according to the modified file items;

if the target file item operation is a file item deleting operation and the scanning information of the deleted file item is in the scanning list, judging whether the file item is migrated or not; if not, deleting the scanning information of the deleted file item from the scanning list; if so, deleting the scanning information of the deleted file item from the scanning list, and writing the operation information of the file item deleting operation into an increment operation list of the target file storage system;

if the target file item operation is a file renaming operation and the scanning information of the renamed file is in the scanning list, judging whether the renamed file is migrated or not; if not, modifying the scanning information of the renamed file in the scanning list; if so, modifying the scanning information of the renamed file in the scanning list, and writing the operation information of the file renaming operation into an increment operation list of the target file storage system;

11. The method of claim 10, further comprising:

reading a second preset number of target to-be-synchronized operation information from the operation information in the incremental operation list according to the latest operation time sequence of the file items;

executing synchronous operation on target file items to be synchronized corresponding to the target file information to be synchronized in the target file storage system according to the target file information to be synchronized;

and returning to execute the acquisition operation of the target information to be synchronized and scanned until all the file items to be synchronized corresponding to the operation information in the incremental operation list execute the synchronization operation.

12. The method according to any one of claims 2-5 and 6-7, wherein the attribute information of the file item at least comprises: the index number of the file entry, the name of the file entry, the parent directory index number of the file entry, and the type of the file entry.

13. The method of claim 1, wherein the target file storage system comprises: an unstructured data management system.

14. A file data migration apparatus configured to an original file storage system, the apparatus comprising:

a migration module, configured to migrate the file items to be migrated, which correspond to the scanning information in the scanning list, to a target file storage system, where the scanning information includes: the inventory scan information and the incremental scan information.

15. A computer device, characterized in that the computer device comprises:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the memory stores a computer program executable by the at least one processor and an original file storage system, the computer program being executable by the at least one processor to enable the at least one processor to perform the file data migration method of any one of claims 1-13.

16. A computer-readable storage medium storing computer instructions for causing a processor to perform the file data migration method of any one of claims 1 to 13 when executed.