CN107943412B

CN107943412B - Method, device and system for splitting partition and deleting data file in partition

Info

Publication number: CN107943412B
Application number: CN201610891411.3A
Authority: CN
Inventors: 杨恋; 杨皓然; 赵波
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2016-10-12
Filing date: 2016-10-12
Publication date: 2021-10-22
Anticipated expiration: 2036-10-12
Also published as: CN107943412A

Abstract

The application discloses a partition splitting method and device, a method and device for deleting data files in a partition and a partition management system. The partition splitting method comprises the following steps: acquiring a data file of a first partition; acquiring a data file of the newly-built partition; and linking the data file of the new partition with the data file of the first partition so as to create a link relation between the new partition and the first partition. According to the technical scheme, the existing reference file is replaced by creating the link relation, so that the problems that the partition splitting time is related to the number of files, the splitting process is complicated, and the partition splitting efficiency is low are solved; particularly, if the user requests sudden increase, the link relation is established between the newly-built partition and the first partition, so that the newly-built partition can be continuously split for multiple times.

Description

Method, device and system for splitting partition and deleting data file in partition

Technical Field

The application relates to the technical field of communication information, in particular to a partition splitting method, a method for deleting data files in partitions and a partition management system; the application also relates to a device for splitting the partition and a device for deleting the data file in the partition.

Background

With the huge increase of data volume due to the development of communication technology, big data has become the focus of today. Among the technologies required for big data, a distributed file system, a distributed database, and the like are all technologies applicable in the context of big data.

Under the condition of large data volume in a distributed file system, the distributed NOSQL database has very high read-write performance, which benefits from the non-relationship and simple structure of the database. In the distributed NOSQL database, one partition is loaded by only one machine. However, as the user requests increase, one partition cannot meet the service requirements, and at this time, the partition needs to be split into 2 or more partitions to meet the service requirements, but the service cannot be provided in the process of splitting the partition, and the access requests of the user all fail. The partition splitting is to drop the original partition after splitting one partition into a plurality of partitions, and make the service of the split new partition on line, so that the writing pressure of the original partition is distributed to different partitions, and the limit of the processing capacity of a single partition is broken through. Currently, the automatic splitting scheme of the NOSQL database is: and creating a reference file for each data file during partition splitting, writing splitting points into the files, and completing the partition splitting after the reference files of all the data files are created. When a new partition is loaded, the reference files are converted into real data files one by one. The new partition cannot be split again until all reference files are replaced with real data files.

Therefore, under the partition splitting scheme of the conventional NOSQL database, the following problems exist: partition splitting time is related to the number of files, a reference file is required to be created for each data file, IO exists when the reference file is created for each data file, splitting process is complicated, the reference files are converted into real data files one by one during loading, IO needs to be performed for multiple times, and partition splitting efficiency is low. In particular, if a user requests a bump, the partition may need to be split multiple times, and cannot be split when reference files are also contained within the partition.

Disclosure of Invention

The application provides a partition splitting method, a method for deleting data files in partitions and a partition management system, so as to solve the problems in the prior art. The application also relates to a device for splitting the partition and a device for deleting the data file in the partition.

The application provides a partition splitting method, which comprises the following steps:

acquiring a data file of a first partition;

acquiring a data file of the newly-built partition;

and linking the data file of the new partition with the data file of the first partition so as to create a link relation between the new partition and the first partition.

Optionally, the data file includes: physical files and map link files.

Optionally, the linking the data file of the new partition with the data file of the first partition includes:

and pointing the mapping link file in the newly-built partition to the data file in the first partition.

Optionally, before the step of linking the data file of the newly created partition with the data file of the first partition, the method includes:

judging whether the data file in the first partition is a mapping link file;

if so, pointing the mapping link file in the newly-built partition to the previous-level data file pointed by the data file in the first partition so as to create a link relation between the newly-built partition and the previous-level partition;

and if not, executing the step of linking the data file of the newly-built partition with the data file of the first partition.

Optionally, the creating a link relationship between the new partition and the first partition includes:

creating a mapping link set in the newly-built partition; the set of mapping links includes: mapping link files in the newly-built partition;

and creating a mapping link pointing to the first partition where the data file pointed by the mapping link file in the mapping link set is located for the newly-built partition.

Optionally, after the step of creating a mapping link set in the new partition, the method includes:

and executing a persistence operation on the mapping link set.

Optionally, after the step of creating, for the new partition, a mapping link pointing to the first partition where the data file pointed by the mapping link file in the mapping link set is located, the method includes:

sending feedback information of the first partition completing the splitting to the main service process;

receiving a partition scheduling request sent by the main service process;

blocking access requests to the split partitions and joining the newly created partitions into the database system.

Optionally, before the step of obtaining the data file of the first partition, the method includes:

and creating at least two new partitions for the first partition.

Optionally, the creating at least two new partitions for the first partition includes:

traversing load pressures of each storage server connected in the database system;

and acquiring a storage server with the load pressure lower than a preset threshold value, and creating two newly-built partitions for the first partition in the storage server.

Optionally, after the step of linking the data file of the new partition with the data file of the first partition to create the link relationship between the new partition and the first partition, the method includes:

creating a data file in the newly-built partition; the data file comprises a physical file.

Correspondingly, this application still provides a device that subregion splits, device that subregion splits includes:

a first data file acquisition unit for acquiring a data file of a first partition;

a new data file obtaining unit, configured to obtain a data file of a new partition;

and the link relation creating unit is used for linking the data file of the newly-built partition with the data file of the first partition so as to create the link relation between the newly-built partition and the first partition.

Optionally, the data file obtaining unit is specifically configured to obtain a physical file and a mapping link file of the first partition.

Optionally, the link relationship creating unit is specifically configured to point the mapping link file in the newly created partition to the data file in the first partition.

Optionally, the apparatus for partitioning further includes:

a mapping link file determining subunit, configured to determine whether the data file in the first partition is a mapping link file before the data file of the newly created partition is linked with the data file of the first partition;

a mapping link pointing subunit, configured to receive a determination result of the mapping link file determining subunit, and if the determination result is positive, point the mapping link file in the new partition to a previous-level data file pointed by the data file in the first partition, so as to create a link relationship between the new partition and the previous-level partition;

and the mapping link relation creation triggering subunit is used for receiving the judgment result of the mapping link file judgment subunit, and if not, triggering the link relation creation unit.

Optionally, the link relationship creating unit further includes:

a set creating subunit, configured to create a mapping link set in the newly created partition; the set of mapping links includes: mapping link files in the newly-built partition;

and the partition mapping creation subunit is used for creating a mapping link pointing to the first partition where the data file pointed by the mapping link file in the mapping link set is located for the newly-built partition.

Optionally, the link relationship creating unit further includes:

and the persistence subunit is configured to perform a persistence operation on the mapping link set after the mapping link set is created in the newly created partition.

Optionally, the apparatus for partitioning further includes:

a feedback information sending unit, configured to send feedback information that the splitting of the first partition is completed to the main service process after creating, for the newly-created partition, a mapping link that points to the first partition where the data file that the mapping link file points to in the mapping link set is located;

a scheduling request receiving unit, configured to receive a partition scheduling request sent by the main service process;

and the partition up-down unit is used for preventing access requests to the split partitions and connecting the newly-built partitions into the database system.

Optionally, the apparatus for partitioning further includes:

and the partition new building unit is used for creating at least two new partitions for the first partition before the data file of the first partition is acquired.

Optionally, the partition new creating unit includes:

a load pressure traversing subunit, configured to traverse load pressures of the storage servers connected in the database system;

and the partition creating subunit is used for acquiring the storage server with the load pressure lower than a preset threshold value, and creating two newly-built partitions for the first partition in the storage server.

Optionally, the apparatus for partitioning further includes:

a physical file creating unit, configured to link the data file of the new partition with the data file of the first partition, so as to create a data file in the new partition after creating a link relationship between the new partition and the first partition; the data file comprises a physical file.

In addition, the present application also provides a method for deleting a data file in a partition, where the method for deleting a data file in a partition includes:

receiving an instruction for deleting a data file;

judging the type of the data file;

if the type of the data file is a physical file, deleting the data file in a partition for storing the data file;

and if the type of the data file is a mapping link file, deleting the data file from a mapping link set of a partition storing the data file.

Optionally, after the step of deleting the data file from the mapping link set of the partition storing the data file, the method includes:

judging whether the mapping link set has mapping link files pointing to other data files in the partition where the upper-level data file pointed by the deleted data file is located;

if not, deleting the mapping link of the partition of the deleted data file pointing to the partition of the upper-level data file pointed by the deleted data file.

Optionally, after the step of deleting the mapping link of the partition from which the data file has been deleted and pointing to the partition in which the upper-level data file pointed by the deleted data file is located, the method includes:

judging whether a mapping link pointing to a partition where a previous-level data file pointed by the deleted data file is located exists in a main service process;

and if not, deleting the data files stored in the partitions.

Correspondingly, the present application also provides a device for deleting a data file in a partition, where the device for deleting a data file in a partition includes:

a deletion instruction receiving unit configured to receive an instruction to delete a data file;

the type judging unit is used for judging the type of the data file;

a physical file deleting unit, configured to receive a determination result of the type determining unit, and delete the data file in the partition in which the data file is stored if the type of the data file is a physical file;

and the mapping link file deleting unit is used for receiving the judgment result of the type judging unit, and deleting the data file in the mapping link set of the partition storing the data file if the type of the data file is the mapping link file.

Optionally, the apparatus for deleting the data file in the partition further includes:

the partition judging unit is used for judging whether the mapping link set has mapping link files pointing to other data files in the partition where the upper-level data file pointed by the deleted data file is located or not after the data file is deleted from the mapping link set of the partition storing the data file;

and the partition link deleting unit is used for receiving the judgment result of the partition judging unit, and if not, deleting the mapping link of the partition in which the deleted data file points to the partition in which the upper-level data file points to the deleted data file is located.

a mapping link determining unit, configured to determine, after the deleting of the mapping link of the partition where the data file is deleted points to the partition where the previous-level data file pointed to by the deleted data file is located, whether a mapping link pointing to the partition where the previous-level data file pointed to by the deleted data file is located exists in a host service process;

and the data file deleting unit is used for receiving the judgment result of the mapping link judging unit, and if not, deleting the data file stored in the partition.

In addition, the present application also provides a partition management system, including:

the apparatus of any one of the above described zone splitting; and

the apparatus for deleting data files in a partition according to any of the above.

Compared with the prior art, the method has the following advantages:

according to the partition splitting method and device, the method and device for deleting the data files in the partitions and the partition management system, the data files of the first partition are obtained; acquiring a data file of the newly-built partition; and linking the data file of the new partition with the data file of the first partition so as to create a link relation between the new partition and the first partition. According to the technical scheme, the existing reference file is replaced by creating the link relation, so that the problems that the partition splitting time is related to the number of files, the splitting process is complicated, and the partition splitting efficiency is low are solved; particularly, if the user requests sudden increase, the link relation is established between the newly-built partition and the first partition, so that the newly-built partition can be continuously split for multiple times.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art according to the drawings.

FIG. 1 shows a flow diagram of a method of partition splitting provided in accordance with an embodiment of the present application;

FIG. 2 is a flowchart illustrating creating a mapping link file pointing to the data file in the newly created partition according to an embodiment of the present application;

FIG. 3 is a diagram illustrating a primary data file pointed to by a map link file in the newly created partition according to an embodiment of the present application;

FIG. 4 is a flowchart illustrating a partition creation linking relationship provided according to an embodiment of the present application for associating the newly created partition with the data file;

FIG. 5 is a flowchart illustrating creating at least two new partitions for the first partition according to an embodiment of the present application;

FIG. 6 illustrates a flow diagram for scheduling a split partition and a new partition provided in accordance with an embodiment of the present application;

FIG. 7 shows a schematic diagram of an apparatus for zone splitting provided in accordance with an embodiment of the present application;

FIG. 8 illustrates a flow diagram of a method of deleting a data file in a partition provided in accordance with an embodiment of the present application;

FIG. 9 illustrates a flow diagram for deleting a mapping link of a partition provided in accordance with an embodiment of the present application;

FIG. 10 illustrates a flow diagram for deleting data files stored in a partition provided in accordance with an embodiment of the present application;

FIG. 11 is a diagram illustrating an apparatus for deleting data files in a partition according to an embodiment of the present application;

FIG. 12 illustrates a schematic diagram of a partition management system provided in accordance with an embodiment of the present application.

Detailed Description

In order that the above objects, features and advantages of the present application can be more clearly understood, the present application will be described in further detail with reference to the accompanying drawings and detailed description. It should be noted that the embodiments and features of the embodiments of the present application may be combined with each other without conflict.

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. This application is capable of implementation in many different ways than those herein set forth and of similar import by those skilled in the art without departing from the spirit and scope of this application, and it is therefore not limited to the specific implementations disclosed below.

The embodiment of the application provides a partition splitting method, a method for deleting data files in a partition and a partition management system; the application also relates to a device for splitting the partition and a device for deleting the data file in the partition. Details are described in the following examples one by one.

Currently, the automatic splitting scheme of the NOSQL database is: and creating a reference file for each data file during partition splitting, writing splitting points into the files, and completing the partition splitting after the reference files of all the data files are created. When a new partition is loaded, the reference files are converted into real data files one by one. The new partition cannot be split again until all reference files are replaced with real data files. Therefore, under the partition splitting scheme of the conventional NOSQL database, the following problems exist: partition splitting time is related to the number of files, a reference file is required to be created for each data file, IO exists when the reference file is created for each data file, splitting process is complicated, the reference files are converted into real data files one by one during loading, IO needs to be performed for multiple times, and partition splitting efficiency is low. In particular, if a user requests a bump, the partition may need to be split multiple times, and cannot be split when reference files are also contained within the partition. Aiming at the problem, according to the technical scheme, by the method for creating the link relation between the newly-created partition and the first partition, data files do not need to be copied in the whole process, the unserviceable time is greatly shortened, the existing reference files are replaced by the created link relation, the newly-created partition can be continuously split for multiple times, and therefore the functions that the partition can be rapidly split and the split partition can be continuously split for multiple times are achieved.

Before describing the specific steps of this embodiment in detail, in order to facilitate understanding of this technical solution, a brief description is made on the NOSQL database.

NOSQL (not Only SQL) database refers to a non-relational database, and is a typical distributed database. The generation of the non-relational database aims to solve the challenges brought by large-scale data set multiple data types, especially the big data application problem. Distributed database systems typically use smaller computer systems, each of which may be individually located in a single location, each of which may have a complete copy, or a partial copy, of the DBMS, and its own local database, with many computers located at different locations interconnected via a network to form a complete, globally logically centralized, physically distributed, large database. The NOSQL database has four major categories as follows: Key-Value (Key-Value) stores databases, which use a hash table with a specific Key and a pointer pointing to specific data. The Key/value model has the advantages of simplicity and easiness in deployment for an IT system; the column storage database is generally used for dealing with mass data stored in a distributed manner. Keys still exist but they are characterized by pointing to multiple columns. The columns are arranged by a family of columns; a document-type database, which is similar to the first type of key-value store. This type of data model is a versioned document, which is stored in a specific format, such as JSON; the graphic (Graph) database, which is different from other rows and columns and SQL database with rigid structure, uses a flexible graphic model and can be expanded to a plurality of servers. NOSQL databases do not have a Standard Query Language (SQL), and therefore data modeling is required to perform database queries. Many NOSQL databases have REST-style data interfaces or query APIs.

The embodiment of the application provides a partition splitting method, which aims to solve the problems of long unserviceable time and multiple splitting requirements in a large-partition splitting scene of multiple files. The embodiment of the partition splitting method is as follows:

referring to fig. 1, a flow chart of a method of partition splitting provided according to an embodiment of the present application is shown.

The partition splitting method comprises the following steps:

step S101, acquiring a data file of a first partition.

In this embodiment, the obtaining of the data file of the first partition may be implemented by: and acquiring a file list of the first partition, and acquiring all data files recorded in the first partition from the file list of the first partition.

For example: after the file list of the first partition B is obtained, the data files B1, B2, B3 and B4 stored in the first partition B are obtained from the file list of the first partition B.

It should be noted that, compared with storing data in the same node, the NOSQL database needs to store data files in a partitioned manner, and disperse the data files over multiple nodes. And typically copies the data file while partitioning. Therefore, the parallel performance is improved, and the problem of single-point failure is solved. In this embodiment, the first partition is a large partition in which multiple files are stored in a NOSQL database. Among them, partition splitting is an important technique in distributed systems, usually considered together with auto-migration and load balancing. The splitting makes one partition become a plurality of sub-partitions, and the data files in the original partition are all owned by each sub-partition. Auto-fragmentation can solve the problem of access to a partition exceeding the processing power of a single machine (storage server).

In this embodiment, the data files at least include two files: physical files and map link files. Wherein the physical file is a data file actually stored in a partition in the NOSQL database, for example: in a Key Value (Key-Value) storage database, a data file is composed of keys (keys) and values (values), wherein logically the keys and the values are in a one-to-one relationship, the keys are equivalent to indexes of the data file, and the values are contents of the data file represented by the keys; the mapping link file (linkfile) is a mapping link file in the current partition pointing to the data file in the original partition before the split, and the mapping link file is not a real physical file but exists logically.

It can be understood that the physical file is an actual file which exists actually, and is a data file which is written into and stored in the partition after the partition is created; the mapping link file is a mapping file pointing to a physical file, when a first partition is split into two new partitions, all data files stored in the first partition are allocated to the two new partitions, because the data files are allocated to the new partitions, the mapping link file pointing to the data files in the first partition is created in the new partitions instead of copying the data files to the new partitions, and when the data files in the partitions which are already split are accessed, the actual storage addresses of the data files are accessed through the mapping link file pointing to the data files in the new partitions.

It should be noted that, since each partition has an identifier of an access range, when a new partition is created, the identifier of the access range needs to be created for the new partition. For example: data files B1, B2, B3 and B4 are stored in the first partition B, the access range is [ begin, end ], after two new partitions C, D are created for the first partition B, the data files B1, B2, B3 and B4 stored in the first partition B are respectively linked to the partition C, D, the split point of the access range of the first partition B is mid, the access range of the new partition C is [ begin, mid), the access range of the new partition D is [ mid, end), when the data files B1 to B4 are accessed through the partition C, only the data files with the access range [ begin, mid) can be read out, and similarly, when the data files B1 to B4 are accessed through the partition D, only the data with the access range [ mid, end) can be read out.

Step S103, acquiring the data file of the newly-built partition.

In this embodiment, the new partition means: the first partition, when split, provides at least two new partitions for the first partition. When the new partition is created, data files with the same quantity are created according to the quantity of the data files in the first partition, and the created data files are mapping link files.

Specifically, creating equal number of data files may be implemented as follows: and acquiring a file list of the first partition, and creating mapping link files with equal quantity according to the number of the data files in the first partition recorded in the file list.

For example: the number of data files B1, B2, B3 and B4 in the first partition B recorded in the file list of the first partition B is 4, and then 4 mapping link files C1, C2, C3, C4, D1, D2, D3 and D4 are respectively created in the newly created partition C and the newly created partition D which are split up.

Step S105, linking the data file of the new partition with the data file of the first partition to create a link relation between the new partition and the first partition.

In this embodiment, the linking of the data file of the new partition and the data file of the first partition may be implemented as follows: in the new partition, a link relationship is created between all data files recorded in the first partition and the data files of the new partition, which are acquired from the file list of the first partition in step S101.

In a specific implementation, the linking of the data file of the new partition with the data file of the first partition is to point, in the new partition, all mapping link files in the new partition to all data files recorded in the first partition, which are obtained from the file list of the first partition in step S101, respectively.

It should be noted that, in this embodiment, when a partition is split, two new partitions are provided for the first partition, so in this step, when creating a link relationship between the new partitions and the first partition, all the data files recorded in the first partition, which are obtained from the file list of the first partition in step S101, are allocated to two new partitions provided for the first partition, a mapping link pointing to a corresponding data file is created for all the data files allocated to the new partitions in each new partition, and an identifier of an access range is created for each new partition.

The partition splitting method is described below by a specific example, for example: having a first partition B, acquiring the file list of the first partition B, then acquiring the data files B1, B2, B3 and B4 stored in the first partition B from the file list of the first partition B, and the access range of the first partition B is [1,100], B1, B2, B3 and B4 all contain 1-100 data, when the first partition B is split, newly created partitions C and D are created in the database, and in this step, the data files B1, B2, B3, B4 are assigned to the new partition C, the data files B1, B2, B3, B4 are assigned to the new partition D, the split point of the access range of the first partition B is 50 because the first partition B is split into two partitions, and creates an access range of [1,50] in the newly created partition C and an access range of [50,100] in the newly created partition D.

If the first partition B stores data files B1, B2, B3, and B4 and the range is [1,100], the first partition B is split into two partitions, so the split point of the access range of the first partition B is 50, the access range of the newly created partition C is [1,50], and the access range of the newly created partition D is [50,100], and when 20 lines of data are accessed, B1-B4 are read through the partition C.

It can be understood that, after the data file of the first partition is allocated to the two new partitions, since an identifier of an access range is created for each new partition, and although each new partition creates a mapping link pointing to the data file in the first partition, the new partitions are accessed according to the access identifiers, since the access ranges of the two new partitions are independent from each other, and the access ranges are not overlapped, the load pressure when the partitions are accessed can be reduced. In this embodiment, two new partitions are provided for the first partition when the partitions are split, but it is not excluded that three or more new partitions are provided for the first partition when the partitions are split in order to reduce the load borne by the partitions, and when three or more new partitions are provided for the first partition, it is also necessary to satisfy that the partitions are independent from each other, and the access ranges are not overlapped.

Since the data files stored in the first partition include both physical files and mapping link files, when step S105 is executed to link the data files of the new partition with the data files of the first partition, that is: when the step of pointing the mapping link file in the new partition to the data file in the first partition is executed, it is necessary to first determine the type of each specific data file, and before the step of linking the data file of the new partition and the data file of the first partition in step S105, the steps specifically include steps S104-1 to S104-3, which are further described below with reference to fig. 2.

Referring to fig. 2, a flowchart of creating a mapping link file pointing to the data file in the new partition according to an embodiment of the present application is shown.

The step of creating the mapping link file pointing to the data file in the new partition comprises the following steps:

step S104-1, judging whether the data file in the first partition is a mapping link file.

In this embodiment, the determining whether the data file in the first partition is a mapping link file may be implemented by: and acquiring a mapping link set of the first partition, and when a mapping link pointing to each corresponding data file is created for each data file allocated to the new partition in the new partition, comparing each data file with the acquired mapping link set of the first partition, and judging whether the data file is recorded in the mapping link set of the first partition.

It should be noted that the mapping link set of the first partition is: a link map file. The link map file is persisted in the manifest file, and in particular, the link map is used for storing the mapping link file in the first partition.

It is to be understood that if the first partition is an original partition, that is: the first partition is not split by other partitions; or, if the partition pointed by the first partition is deleted, then there is no mapping link set in the first partition or the mapping link set in the first partition is empty, and at this time, it indicates that the data files in the first partition are all physical files.

Following the specific examples described above: judging the types of the data files B1, B2, B3 and B4, because B1 and B2 are in the mapping link set of the partition B, the data files B1 and B2 are explained to be mapping link files, and B3 and B4 are physical files if B3 and B4 are not in the mapping link set of the partition B.

And step S104-2, if yes, pointing the mapping link file in the new partition to the previous-level data file pointed by the data file in the first partition so as to create a link relation between the new partition and the previous-level partition.

In this embodiment, this step receives the determination result in step S104-1, and if the piece of data file is recorded in the mapping link set of the first partition, step S104-2 is entered and executed to point the mapping link file in the new partition to the previous level data file pointed by the data file in the first partition, so as to create a link relationship between the new partition and the previous level partition.

The mapping link file in the new partition points to the previous-level data file pointed by the data file in the first partition to create the link relationship between the new partition and the previous-level partition, which can be implemented in the following manner: and acquiring the last-level data file pointed by the data file from the mapping link set of the first partition, so that the acquired mapping link file in the current newly-built partition points to the last-level data file pointed by the data file acquired from the mapping link set of the first partition.

Following the above specific example and referring to fig. 3, a schematic diagram of the previous level data file pointed to by the mapping link file in the new partition according to the embodiment of the present application is shown: acquiring mapping links (B1 → A1, B2 → A2) of the data files B1 and B2 of the partition B from the mapping link set of the first partition B, enabling the mapping link file C1 in the new partition C to point to the mapping link file C1(C1 → A1) of the data file A1 pointed by the mapping link file B1 allocated to the new partition C in the new partition C, and enabling the mapping link file C2 in the new partition C to point to the mapping link file C2(C2 → A2) of the data file A2 pointed by the mapping link file B2 allocated to the new partition C in the new partition C; in the new partition D, the mapping link file D1 in the new partition D is made to point to the mapping link file D1(D1 → A1) of the data file A1 pointed by the mapping link file B1 allocated to the new partition D, and the mapping link file D2 in the new partition D is made to point to the mapping link file D2(D2 → A2) of the data file D2 pointed by the mapping link file B2 allocated to the new partition D.

It can be understood that, since the map link file is not an actually existing physical file and is a map pointing to an actually existing physical file, when the data file in the first partition is a map link file, the map link file created in the new partition needs to point to a previous data file pointed by the map link file in the first partition until the map link file created in the new partition points to a physical file.

And step S104-3, if not, executing the step of linking the data file of the newly-built partition with the data file of the first partition.

In this embodiment, the step receives the determination result in the step S104-1, and if the piece of data file is not recorded in the mapping link set of the first partition, the process proceeds to the step S104-3 and the step of linking the data file of the newly created partition and the data file of the first partition is executed.

The creating of the mapping link file pointing to the data file in the new partition can be implemented in the following manner: and in the current new partition, pointing the mapping link file in the new partition to the data file allocated to the new partition.

Following the specific examples described above: causing the mapping link file C3 in the newly created partition C to point to the mapping link file C3 of the storage address of the physical file B3 (C3 → B3); causing the mapping link file C4 in the newly created partition C to point to the mapping link file C4 of the storage address of the physical file B4 (C4 → B4); causing the mapping link file D3 in the new partition D to point to the mapping link file D3 of the storage address of the physical file B3 (D3 → B3); causing the mapping link file D4 in the new partition C to point to the mapping link file D4 of the storage address of the physical file D4(D4 → B4);

after the data file is linked with the newly created partition, the newly created partition needs to create a link relationship with the first partition, and the creating of the link relationship between the newly created partition and the partition associated with the data file specifically includes the following steps: steps S103-4 to S103-5, which will be further described with reference to fig. 4.

Referring to fig. 4, a flowchart of creating a link relationship between the newly created partition and the partition associated with the data file according to an embodiment of the present application is shown.

The creating of the link relation between the new partition and the partition associated with the data file comprises the following steps:

step S103-4, a mapping link set is created in the newly-built partition; the set of mapping links includes: and mapping the link file in the new partition.

In this embodiment, the creating of the mapping link set in the newly created partition may be implemented by the following method: and creating a mapping link set in the newly-built partition, traversing and acquiring all mapping link files pointing to data files created in the newly-built partition, and writing all the acquired mapping link files into the mapping link set. It should be noted that the mapping link set of the first partition is: a link map file.

It can be understood that, in the step of creating a mapping link file pointing to a data file in a newly created partition, a mapping relationship between each specific data file is created, each mapping link file is independent from other mapping link files, and the mapping link files are numerous and scattered and are not suitable for maintenance, so in this step, the mapping relationship between each specific data file is stored in a link map file, that is: and taking the link map file as a directory file for mapping the link file in the current new partition, recording the data range of the current new partition, and finishing the establishment of the file link relation in the new partition.

It can be understood that, since the mapping relationship between each specific data file is stored in one link map file in the newly-built partition, the link map file can be completed in one IO when being persisted, and the length of time that the link map file cannot be served is reduced.

Following the specific examples described above: a set of map links map is created in the new partition C, which records { C1 → A1, C2 → A2, C3 → B3, C4 → B4}, and in the new partition D, which records { D1 → A1, D2 → A2, D3 → B3, D4 → B4 }.

Step S103-5, creating a mapping link pointing to the first partition where the data file pointed by the mapping link file in the mapping link set is located for the newly-built partition.

In this embodiment, after the mapping link relationship between the data file levels is established, a link relationship between partitions also needs to be established, and the creating of the mapping link pointing to the first partition where the data file pointed by the mapping link file in the mapping link set is located for the newly-created partition may be implemented in the following manner: and creating a mapping link of the partition where the data file pointed by the mapping link file in the mapping link set pointed by the current new partition to the new partition in the main service process. For example: if the first partition is B and the current newly-built partition is C, in the step, C- > { B } is created in the main service process, and the first partition B is dependent on a partition mapping link of the newly-built partition C; and if the first partition B also depends on the partition A, creating C- > { A, B } in the main service process, wherein the first partition B and the partition A depend on the partition mapping link of the newly-built partition C.

Because the mapping link determining dependency relationship is directly created when the mapping link of the partition where the data file pointed by the mapping link file in the mapping link set pointed by the current new partition is created in the main service process, the step can also be completed in one IO.

It should be noted that the main service process is a common role in the NOSQL database, and the main server process is called a master in the NOSQL database, and the master is mainly responsible for partition scheduling; corresponding to the master, the NOSQL database also comprises a worker role, and the worker is mainly responsible for partition read-write operation.

For example: creating partition mapping links C → { A, B } (partition C points to partitions A, B, partition A, B are depended on by partition C), D → { A, B } (partition D points to partitions A, B, partition A, B are depended on by partition D) for newly created partitions C and D in the master role in the NOSQL database.

As a preferred embodiment, in the partition splitting method provided in this embodiment of the present application, since partitions are loaded in storage servers (machines), when an access received by a partition exceeds the processing capacity of a single storage server, in order to reduce the load pressure of the storage server that loads the partition, at least two new partitions are created for the partition in an NOSQL database, so that the state of the partition is in a to-be-split state, and step S101 is triggered to acquire a data file of a first partition, so as to split the partition.

The creating of at least two new partitions for the first partition specifically includes steps S100-1 to S100-2, which will be further described with reference to fig. 5.

Referring to fig. 5, a flowchart for creating at least two new partitions for the first partition according to an embodiment of the present application is shown.

The step of creating at least two newly-built partitions for the first partition comprises the following steps:

step S100-1, the load pressure of each storage server connected in the database system is traversed.

In this embodiment, the traversing the load pressure of each storage server connected in the database system may be implemented as follows: and determining each storage server which is connected in the NOSQL database system at present, and acquiring the load pressure of each storage server which is connected in the NOSQL database system at present.

For example: the number of the connected storage servers in the NOSQL database system is 5, wherein the state of a partition loaded in one storage server is in a to-be-split state, and the load pressure of other 4 storage servers connected in the NOSQL database system at the current moment is obtained.

And S100-2, acquiring a storage server with the load pressure lower than a preset threshold value, and creating two newly-built partitions for the first partition in the storage server.

In this embodiment, the obtaining of the storage server with the load pressure lower than the preset threshold may be implemented as follows: comparing the load pressure of each storage server connected in the NOSQL database system at the current time acquired in the step S100-1 with a preset threshold value, and acquiring the storage server with the load pressure lower than the preset threshold value.

For example: after comparing the load pressure with a preset threshold value, acquiring 1 connected storage server in the NOSQL database system, and creating two newly-built partitions for the first partition in the acquired storage server; and if 2 connected other storage servers are acquired in the NOSQL database system, respectively creating two newly-built partitions for the first partition in the acquired 2 storage servers.

It should be noted that the storage servers acquired in this step are calculated according to load balancing, and according to the load conditions of the storage servers connected in the NOSQL database system, one storage server with a load pressure lower than a preset threshold may be acquired during acquisition, or two storage servers with a load pressure lower than a preset threshold may be acquired.

In this embodiment, the storage servers for creating the newly-built partition may also be obtained by comparing the load pressures of the storage servers, for example: if the load pressure of one storage server in the connected storage servers in the NOSQL database system is greatly lower than that of other storage servers, acquiring the storage server in the step to create two newly-built partitions for the first partition; if the load pressures of the connected storage servers in the NOSQL database system are similar, in order to prevent the load of the storage server from exceeding the processing capacity of the storage server after the partition is created, two storage servers with the lowest load pressures are obtained in the step, and a newly created partition is created in each of the two storage servers.

In the partition splitting method provided in the embodiment of the present application, since the creating of the map link set in the newly created partition is performed in a memory in an NOSQL database system, the map link set has a time limit for transient data, and in order to enable the created map link set to be stored for a long time, the technical solution of this embodiment provides a preferred embodiment, in the preferred embodiment, after the creating of the map link set in the newly created partition in step S103 is performed, a persistence operation is performed on the map link set, that is, a link map file is persisted in a manifest file. The manifest file is a file in a storage NOSQL database system, each partition has a manifest file, and the manifest files of the partitions are not overlapped.

It should be noted that the persistence operation is a mechanism for transferring program data between a persistent state and a transient state. Specifically, transient data (e.g., data in memory that cannot be permanently stored) is persisted as persistent data (e.g., persisted to a database that can be permanently stored).

Following the specific examples described above: creating a link map of the mapping link set in the new partition C, wherein the link map records { C1 → A1, C2 → A2, C3 → B3, C4 → B4}, creating the link map of the mapping link set in the new partition D, wherein the link map records { D1 → A1, D2 → A2, D3 → B3, D4 → B4}, persisting the link map file of the new partition C into the manifest file of C, and persisting the link map file of the new partition D into the manifest file of D.

As a preferred embodiment, in the partition splitting method provided in this embodiment of the present application, after the first partition is split into two new partitions, when accessing a data file originally stored in the first partition, the data file is accessed by reading mapping link files in the two new partitions corresponding to the first partition, instead of directly accessing a partition that has been split, that is, without directly interacting with the partition, so after the step of creating a mapping link pointing to the first partition for the new partition, the split partition and the new partition need to be scheduled, which specifically includes the following steps: steps S106-1 to S106-3 are further described below with reference to FIG. 6.

Referring to fig. 6, a flow chart for scheduling a split partition and a new partition according to an embodiment of the present application is shown.

The scheduling of the split partitions and the newly-built partitions comprises the following steps:

and step S106-1, sending feedback information of the completion of the splitting of the first partition to the main service process.

In this embodiment, the sending of the feedback information that the first partition completes the splitting to the main service process may be implemented by the following method: and sending feedback information of completing splitting for the first partition to the main service process through a communication protocol between a master role and a worker role in the NOSQL database.

It should be noted that, the core idea of the master-worker design mode is to parallelize the original serial logic and split the logic into a plurality of independent modules for parallel execution, wherein the master role and the worker role mainly comprise two main roles, the master is mainly responsible for scheduling and maintaining the worker queue at the same time, and the task is scheduled to the worker for parallel execution; the worker role mainly performs actual logic calculation and returns the result to the master. In this embodiment, because the NOSQL database is applied to the distributed system, since each storage server in the distributed system is distributed at different locations, and the master role and the worker role are also distributed on different storage servers, a data transmission link needs to be established when the master role and the worker role communicate with each other, and the data transmission link is a communication protocol between the master role and the worker role.

It can be understood that, in this step, a mapping link set is created for the newly created partition, and the worker role that creates the mapping link pointing to the first partition communicates with a master service process (master).

And step S106-2, receiving the partition scheduling request sent by the main service process.

In this embodiment, the partition scheduling request includes: a partition online request and a partition offline request. It can be understood that, according to the different roles of the partitions, the partitions receive the corresponding partition scheduling request sent by the main service process. The receiving of the partition scheduling request sent by the main service process may be implemented in the following manner: the method comprises the steps that a partitioned area which is split receives a partitioned off-line request sent by a main service process; and the newly-built partition receives and receives a partition online request sent by the main service process.

And step S106-3, preventing the access request to the split partition, and connecting the newly-built partition into the database system.

In this embodiment, the blocking of the access request to the split partition may be implemented as follows: the split partition executes related work (such as a running environment, a test environment, a development environment and the like) for removing the split partition according to a received partition offline request sent by the main service process; the new partition is connected into the database system, and the method can be realized by adopting the following modes: and the newly-built partition executes preparation work (such as a running environment, a testing environment, a development environment and the like) for deploying the newly-built partition according to the received partition online request sent by the main service process.

It should be noted that blocking access requests to the split partition means: preventing a user from directly initiating an access request to the split partition. It will be appreciated that after the partitioned partition is blocked from access, no processing operations are performed on the data files in the partition. If the split partition is to be accessed, the data file in the split partition can be accessed through the mapping link file in the new partition corresponding to the split partition. After the newly-built partition is connected and enters the database system, the read-write operation of the data file can be executed, and an access request sent by a user to the newly-built partition is received.

As a preferred embodiment, in the partition splitting method provided in this embodiment of the present application, after the step S105 of linking the data file of the new partition with the data file of the first partition to create the linking relationship between the new partition and the first partition is performed, since a user can access the new partition, an operation of reading or writing data can be performed on the new partition, and when a write operation is performed on the new partition, a data file is created in the new partition; the data file comprises a physical file.

The splitting process of the partition splitting method is described in detail below by specific examples.

Since the data file includes a physical file and a mapping link file, the following two cases may be included according to the type of the data file stored in the partition: only the physical files in the partition and only the mapping link files in the partition.

If the partition A only has physical files, the partition A is not split by other partitions; or the partition pointed by the partition A and the mapping link file in the partition A are deleted. Creating newly-built partitions B and C in a database when the partition A is split; acquiring a file list of data files of the partition A, and acquiring physical files A1, A2, A3 and A4 recorded in the partition A from the file list of the partition A, wherein the access range of the partition A is [1,50 ]; allocating physical files A1, A2, A3 and A4 to a new partition B, allocating physical files A1, A2, A3 and A4 to a new partition C, wherein the partition A is divided into two partitions, so that the division point of the access range of the partition A is 25, an identifier of an access range [1,25 ] is created in the new partition B, and an identifier of an access range [25,50] is created in the new partition C; judging the types of the data files a1, a2, A3 and A4, and since a1, a2, A3 and A4 are all physical files, in the new partition B, the mapping link files B1, B2, B3 and B4 in the new partition B are respectively directed to the storage addresses of the physical files a1, a2, A3 and A4 allocated to the new partition B (B1 → a1, B2 → a2, B2 → a2), and in the new partition C, the mapping link files C2, C2 and C2 in the new partition C are respectively directed to the storage addresses of the physical files a2, a2 and a2 allocated to the new partition C (C2, a2, C2 a → 2, C2, a2 → 2, a2 and a2 → 2); creating a link map of a mapping link set in a new partition B, wherein the link map records { B1 → A1, B2 → A2, B3 → A3, B4 → A4}, creating the link map of the mapping link set in a new partition C, wherein the link map records { C1 → A1, C2 → A2, C3 → A3, C4 → A4}, persisting the link map file of the new partition B into the manifest file of B, and persisting the link map file of the new partition C into the manifest file of C; creating partition mapping links B → { A } (partition B points to partition A, partition A is relied on by partition B), C → { A } (partition C points to partition A, partition A is relied on by partition C) of newly created partitions B and C in a master role in the NOSQL database; and dispatching the partition A to be offline, dispatching the newly-built partition B, C to be online, and reading the data A1-A4 only through the partition B when the data between 1 and 24 needs to be accessed because the access range of the newly-built partition B is marked as [1,25 ], and reading the data A1-A4 only through the partition C when the data between 25 and 50 needs to be accessed because the access range of the newly-built partition C is marked as [25,50] in the same way.

It can be understood that, when a partition is split and a data file is linked to a new partition, only a mapping link file is in the new partition, and a physical file is stored in the new partition after a data writing operation is performed on the new partition.

If only mapping link files exist in the partition B, the partition B is split by other partitions, newly-built partitions C and D are created in a database when the partition B is split, a file list of data files of the partition B is obtained, the data files B1, B2, B3 and B4 recorded in the partition B are obtained from the file list of the partition B, and the access range of the partition B is [1,50 ]; allocating data files B1, B2, B3 and B4 to a new partition C, allocating data files B1, B2, B3 and B4 to a new partition D, wherein the partition B is split into two partitions, so that the split point of the access range of the partition B is 25, an identifier of an access range [1,25 ] is created in the new partition C, and an identifier of an access range [25,50] is created in the new partition D; judging the types of the data files B, B and B, acquiring mapping links (B → A ) of the data files B, B and B of the partition B from the mapping link set of the partition B, making the mapping link files C, C and C in the new partition C respectively point to the data files A, A and A (C → A, C → A, C → A, C → A) pointed by the mapping link files B, B and B distributed to the new partition C, making the mapping link files D, D and D in the new partition D respectively point to the data files A, A and A (D → A, D → A, D → A, B → A, d4 → A4); creating a link map of a mapping link set in the new partition C, wherein the link map records { C1 → A1, C2 → A2, C3 → A3, C4 → A4}, creating the link map of the mapping link set in the new partition D, wherein the link map records { D1 → A1, D2 → A2, D3 → A3, D4 → A4}, persisting the link map file of the new partition C into the manifest file of C, persisting the link map file of the new partition D into the manifest file of D; creating partition mapping links C → { A, B } (partition C points to partitions A, B, partition A, B are depended on by partition C), D → { A, B } (partition D points to partitions A, B, partition A, B are depended on by partition D) of newly created partitions C and D in a master role in the NOSQL database; and dispatching the partition B off line, dispatching the newly-built partition C, D on line, and reading the B1-B4 only through the partition C when the data between 1 and 24 needs to be accessed because the access range of the newly-built partition C is marked as [1,25 ], and reading the B1-B4 only through the partition D when the data between 25 and 50 needs to be accessed because the access range of the newly-built partition D is marked as [25,50] in the same way.

It should be noted that only the mapping link file in the first partition indicates that after the partition is online, the data file has not been written yet or the partition needs to be continuously split due to a sudden increase of requests.

According to the partition splitting method provided by the embodiment of the application, the data file is linked into a new partition, so that a link relation is created between the new partition and a partition associated with the data file; the data files do not need to be copied in the whole process, the unserviceable time is greatly shortened, the existing reference files are replaced by creating the link relation, the newly-built partition can be continuously split for many times, and therefore the functions that the partition can be rapidly split and the split partition can be continuously split for many times are achieved.

In the foregoing embodiment, a partition splitting method is provided, and corresponding to the foregoing partition splitting method, the present application also provides a partition splitting apparatus. Since the embodiments of the apparatus are substantially similar to the embodiments of the method, they have been described in a relatively simple manner, and reference may be made to the description of the embodiments of the method for relevant points. The device embodiments described below are merely illustrative. The partition splitting apparatus embodiment is as follows:

please refer to fig. 7, which illustrates a schematic diagram of a partition splitting apparatus provided according to an embodiment of the present application.

The partition splitting device comprises: a data file acquisition unit 701, a new data file acquisition unit 703 and a link relationship creation unit 705;

the data file acquiring unit 701 is configured to acquire a data file of a first partition;

the new data file obtaining unit 703 is configured to obtain a data file of a new partition;

the link relationship creating unit 705 is configured to link the data file of the new partition with the data file of the first partition, so as to create a link relationship between the new partition and the first partition.

Optionally, the data file obtaining unit 701 is specifically configured to obtain a physical file and a mapping link file of the first partition.

Optionally, the link relationship creating unit 705 is specifically configured to point the mapping link file in the newly created partition to the data file in the first partition.

Optionally, the partition splitting apparatus further includes: the mapping link file judgment subunit, the mapping link pointing subunit and the mapping link relation creation triggering subunit are connected in sequence;

the mapping link file judging subunit is configured to, before the data file of the newly created partition is linked with the data file of the first partition, judge whether the data file in the first partition is a mapping link file;

the mapping link pointing subunit is configured to receive a determination result of the mapping link file determining subunit, and if the determination result is positive, point the mapping link file in the new partition to a previous-level data file pointed by the data file in the first partition, so as to create a link relationship between the new partition and the previous-level partition;

Optionally, the link relationship creating unit 703 further includes: a set creation subunit and a partition mapping creation subunit;

the set creating subunit is configured to create a mapping link set in the newly created partition; the set of mapping links includes: mapping a link file in the newly-built partition;

and the partition mapping creation subunit is configured to create, for the newly created partition, a mapping link pointing to the first partition where the data file pointed by the mapping link file in the mapping link set is located.

Optionally, the link relationship creating unit 705 further includes: a persistence subunit;

Optionally, the apparatus for partitioning further includes: the system comprises a feedback information sending unit, a scheduling request receiving unit and a partition online and offline unit;

the feedback information sending unit is configured to send feedback information that the splitting of the first partition is completed to the main service process after creating, for the newly-created partition, a mapping link that points to the first partition where the data file that the mapping link file points to in the mapping link set is located;

the scheduling request receiving unit is used for receiving a partition scheduling request sent by the main service process;

and the partition online and offline unit is used for preventing access requests to the split partitions and connecting the newly-built partitions into the database system.

Optionally, the apparatus for partitioning further includes: a partition newly-built unit;

and the partition newly building unit is used for building at least two newly built partitions for the first partition before the data file of the first partition is obtained.

Optionally, the partition new creating unit includes: the load pressure traversing subunit and the partition creating subunit;

the load pressure traversing subunit is used for traversing the load pressure of each storage server connected in the database system;

and the partition creating subunit is configured to acquire a storage server with a load pressure lower than a preset threshold, and create two newly-created partitions for the first partition in the storage server.

Optionally, the apparatus for partitioning further includes: a physical file creating unit;

the physical file creating unit is configured to link the data file of the new partition with the data file of the first partition, so as to create a data file in the new partition after creating a link relationship between the new partition and the first partition; the data file comprises a physical file.

In the above embodiments, a partition splitting method and a partition splitting device are provided, and in addition, a method for deleting a data file in a partition is also provided in the present application; the embodiment of the method for deleting the data file in the partition is as follows:

referring to fig. 8, shown is a flowchart of a method for deleting a data file in a partition according to an embodiment of the present application.

The method for deleting the data file in the partition comprises the following steps:

in step S801, an instruction to delete a data file is received.

In this embodiment, the receiving of the instruction to delete the data file may be implemented as follows: and receiving a deleting instruction of a user for a specific data file in the NOSQL database, wherein the deleting instruction carries a partition of the specific data file, which is inquired by the user through a key value and stores a corresponding key, in the NOSQL database.

In the NOSQL database, a data file is composed of a Key (Key) and a Value (Value), where logically the Key and the Value are in a one-to-one relationship, the Key corresponds to an index of the data file, and the Value is the content of the data file represented by the Key.

For example: an instruction to delete data file C1, which is stored in partition C1, is received in this step.

Step S803, determine the type of the data file.

In this embodiment, the types of the data file include: physical files and map link files. The determining of the type of the data file may be implemented as follows: and acquiring a mapping link set in a partition for storing the data file, comparing the data file with the acquired mapping link set, and judging whether the data file is recorded in the mapping link set of the partition.

It should be noted that the mapping link set of the partition is: a link map file. The link map file is persisted in the manifest file, and in particular, the link map is used for storing the mapping link file in the partition.

It can be understood that if the mapping link set of the partition is empty, it indicates that the partition is an original partition or the partition pointed to by the original partition has been deleted, the types of the data files stored in the partition are both physical files, and if the mapping link set of the partition is not empty and the data file is recorded in the mapping link set of the partition, it indicates that the data file is a mapping link file; and if the data file is not recorded in the mapping link set of the partition, indicating that the data file is a physical file.

Step S805, if the type of the data file is a physical file, deleting the data file in the partition storing the data file.

In this embodiment, since the data file is a physical file stored in a partition, the data file may be deleted in the partition storing the data file by a deleteddirectory () method.

In step S807, if the type of the data file is a mapping link file, the data file is deleted from the mapping link set of the partition storing the data file.

In this embodiment, the deleting the mapping link file from the mapping link set of the partition storing the data file may be implemented as follows: and deleting the mapping link file from the link map file which is persisted in the manifest file.

The following describes a process flow of the present embodiment with a specific example.

The data file C1 is stored in the partition C, and the partition C also stores data files C2, C3, C4 and C5, wherein the data files C1, C2 and C3 have the following characteristics for the map link files stored in the map link set link map: (C1 → a1), (C2 → a2), (C3 → B1), the data files C4, C5 are physical files, and the mapping of the partition stored in the read host service process is linked with: { C → A }, { C → B }, { D → B }, that is: in this step, since the data file C1 is a map link file, the map link file is deleted from the link map file persisted in the manifest file (C1 → a1), and the map link set link map is modified to be (C2 → a2), (C3 → B1).

In the method for deleting a data file in a partition provided in the embodiment of the present application, after a mapping link file is deleted in a mapping link set link map, a mapping link between partitions may change, so that a dependency relationship between partitions changes, and thus the technical scheme of the embodiment provides a preferred implementation manner, in which, in the preferred implementation manner, after the mapping link file is deleted in a mapping link set storing partitions of the data file in step S807, the mapping link of a partition needs to be deleted, which specifically includes the following steps: steps S808-1 to S808-2 are further described below with reference to FIG. 9.

Refer to FIG. 9, which illustrates a flowchart of deleting a mapping link of a partition according to an embodiment of the present application.

The deleting the mapping link of the partition comprises the following steps:

step S808-1, judging whether the mapping link set has mapping link files pointing to other data files in the partition where the upper-level data file pointed by the deleted data file is located.

In this embodiment, the determining whether the mapping link set has a mapping link file pointing to another data file in the partition where the previous-level data file pointed by the deleted data file is located may be implemented in the following manner: the mapping link file deleted in step S807 is acquired, the partition where the pointed previous-level data file is located is acquired from the mapping link file, the partition is compared with the mapping link set, and it is determined whether there is any mapping link file pointing to the partition in the mapping link set.

It should be noted that, the partition pointed to by the mapping link file is actually the data file pointed to by the partition, but the data file is only stored in the corresponding partition, so that the partition storing the pointed data file can be obtained from the data file. For example: the mapping link file is B1 pointing to A1, A1 being stored only in partition A.

The description follows with the example above: after the map link file C1(C1 → a1) is deleted from the link map file persisted in the manifest file, since the map link file C1 points to the data file a1, it is determined from the map link set of the current partition whether there is any map link file pointing to the partition a (e.g., points to a2, A3, a4, etc.), and since there is any map link file C2(C2 → a2) in the map link set of the current partition, it is indicated that the partition C points to the partition a, so that the map link between the partitions is not changed, and the dependency between the partitions is also not changed.

Step S808-2, if not, deleting the mapping link of the partition from which the data file is deleted and pointing to the partition in which the upper-level data file pointed by the deleted data file is located.

In this embodiment, the determination result in step S808-1 is received, and if the mapping link set does not have mapping link files pointing to other data files in the partition where the previous-level data file pointed to by the deleted data file is located, the step is entered and the deletion of the mapping link of the partition where the data file is deleted pointing to the partition where the previous-level data file pointed to by the deleted data file is located is triggered. The deleting of the mapping link of the partition from which the data file is deleted pointing to the partition in which the upper-level data file pointed by the deleted data file is located may be implemented in the following manner: and deleting the mapping link of the partition in which the deleted data file pointed to by the partition in which the data file is deleted is located from the main service process.

The description follows with the example above: after the data file C1 is deleted, the data file C3 is deleted from the partition C, the method for deleting data files in the partition according to the embodiment is executed again, since the data file C3 is a map link file, the map link file is deleted from the link map file persisted in the manifest file (C3 → B1), so that the map link set link map is modified to (C2 → a2), since the map link file C3 points to the data file B1, it is determined from the map link set of the partition from which the map link file C3 has been deleted that whether there are map link files pointing to the partition B (for example, points to B2, B3, B4, etc.), since there is no map link file pointing to the partition B again in the map link set of the current partition C, it is said that the partition B no longer depends on the partition C, so the map link between partitions changes, and then the map link of the partition B link pointing to the data file C1 of the data file C3 pointed to by the current partition C is deleted from the main service process Next { C → B }, the mapping link of the partition stored in the host service process is modified to { C → A }, and { D → B }.

Since the dependency relationship between the partitions may be changed after the mapping link { C → B } is deleted from the mapping links of the partitions stored in the main service process, so that the split partitions are no longer dependent on any partition, and the storage space occupied by the redundant data file is reduced, the technical solution of this embodiment provides a preferred implementation manner, in which, after the mapping link of the partition from which the data file is deleted pointing to the partition in which the upper-level data file pointed by the deleted data file is located is deleted in step S808-2, the data file stored in the partition needs to be deleted, which specifically includes the following steps: steps S809-1 to S809-2 are further described below with reference to FIG. 10.

Referring to fig. 10, a flowchart of deleting a data file stored in a partition is shown according to an embodiment of the present application.

The deleting the data files stored in the partition includes:

step 809-1, judging whether a mapping link pointing to the partition where the upper-level data file pointed by the deleted data file is located exists in the main service process.

In this embodiment, the determining whether the mapping link pointing to the partition where the upper-level data file pointed by the deleted data file is located exists in the main service process may be implemented by the following method: acquiring the mapping link of the partition deleted in step S808-2, acquiring the pointed partition from the mapping links of the partition, comparing the partition with the mapping links of the partitions stored in the main service process, and determining whether the mapping link of the partition which has the mapping link deleted exists in the main service process.

The description follows with the example above: after the mapping link { C → B } of the partition B of the data file B1 pointed by the data file C3 pointed by the current partition C is deleted from the main service process, because the mapping link of the partition C is pointed to the partition B, the partition B is compared with the mapping link of the partition stored in the main service process, whether the mapping link pointed to the partition B still exists in the main service process is judged, because the mapping link of the partition stored in the main service process also has the mapping link { D → B }, which indicates that the partition D also points to the partition B, the split partition B is also depended on by the partition D, and the mapping link of the partition pointed to which the mapping link is deleted exists in the main service process, the step S809-2 is not triggered.

And step 809-2, if not, deleting the data files stored in the partitions.

In this embodiment, the determination result in step S809-1 is received, and if there is no mapping link pointing to the partition from which the mapping link has been deleted in the main service process, the step is entered and deletion of the data file stored in the partition is triggered. The deleting of the data file stored in the partition may be implemented in the following manner: deleting the data files stored in the partition through a deleteddirectory () method.

The description follows with the example above: after deleting the data file C3, deleting C2 from the partition C, re-executing the method for deleting the data file in the partition of the embodiment, since the data file C2 is a map link file, deleting the map link file C2(C2 → a2) from the link map file persisted in the manifest file, making the map link set link map null, since the map link file C2 refers to the data file a2, determining whether there is any map link file pointing to the partition a (for example, pointing to A3, a4, etc.) from the map link set of the current partition, since there is no map link file pointing to the partition a again in the map link set of the current partition C, it is indicated that the partition a does not depend on the partition C, so that the map link between the partitions changes, deleting the map link { C → a } of the partition a of the data file a2 pointing to the data file C2 pointed to by the current partition C from the main service process, the mapping link of the partition stored in the main service process is modified to { D → B }, because the mapping link of the partition C is directed to the partition A, the mapping link of the partition A and the mapping link of the partition stored in the main service process are compared, whether the mapping link directed to the partition A still exists in the main service process is judged, and because the mapping link directed to the partition A does not exist in the mapping link of the partition stored in the main service process, the data file stored in the partition A is deleted through a DeleteDirectory () method.

In the foregoing embodiment, a method for deleting a data file in a partition is provided, and the application also provides a device for deleting a data file in a partition, corresponding to the method for deleting a data file in a partition. Since the embodiments of the apparatus are substantially similar to the embodiments of the method, they have been described in a relatively simple manner, and reference may be made to the description of the embodiments of the method for relevant points. The device embodiments described below are merely illustrative. The embodiment of the device for deleting the data file in the partition comprises the following steps:

referring to fig. 11, a schematic diagram of an apparatus for deleting a data file in a partition according to an embodiment of the present application is shown.

The device for deleting the data file in the partition comprises the following steps: a deletion instruction receiving unit 1101, a type determining unit 1103, a physical file deleting unit 1105, and a map link file deleting unit 1107;

the deletion instruction receiving unit 1101 is configured to receive an instruction to delete a data file;

the type determining unit 1103 is configured to determine a type of the data file;

the physical file deleting unit 1105 is configured to receive a determination result of the type determining unit 1103, and delete the data file in the partition storing the data file if the type of the data file is a physical file;

the map link file deleting unit 1107 is configured to receive the determination result of the type determining unit 1103, and delete the data file in the map link set of the partition in which the data file is stored if the data file is a map link file.

Optionally, the apparatus for deleting the data file in the partition further includes: a partition judging unit and a partition link deleting unit;

the partition judging unit is configured to judge whether a mapping link file pointing to another data file in a partition where a previous-level data file pointed by the deleted data file is located exists in the mapping link set after the data file is deleted in the mapping link set of the partition where the data file is stored;

Optionally, the apparatus for deleting the data file in the partition further includes: a mapping link judging unit and a data file deleting unit;

the mapping link judging unit is configured to judge whether a mapping link pointing to a partition where a previous-level data file pointed by the deleted data file is located exists in a host service process after the mapping link pointing to the partition where the previous-level data file pointed by the deleted data file is located is deleted;

In the above embodiments, a partition splitting method, a partition splitting apparatus, a method for deleting a data file in a partition, and a device for deleting a data file in a partition are provided, and in addition, the present application also provides a partition management system; the embodiment of the partition management system comprises the following steps:

referring to fig. 12, a schematic diagram of a partition management system provided according to an embodiment of the present application is shown.

The partition management system comprises: a partition splitting device 1201 and a device 1203 for deleting the data file in the partition;

the device 1201 for partitioning comprises: acquiring a data file of a first partition; linking the data file to a new partition, and enabling the new partition and the partition associated with the data file to establish a link relation;

the apparatus 1203 for deleting the data file in the partition includes: receiving an instruction for deleting a data file; judging the type of the data file; if the type of the data file is a physical file, deleting the data file in a partition for storing the data file; and if the type of the data file is a mapping link file, deleting the data file from a mapping link set of a partition storing the data file.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

1. Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory computer readable media (transient media), such as modulated data signals and carrier waves.

2. As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

Although the present application has been described with reference to the preferred embodiments, it is not intended to limit the present application, and those skilled in the art can make variations and modifications without departing from the spirit and scope of the present application, therefore, the scope of the present application should be determined by the claims that follow.

Claims

1. A method of partition splitting, comprising:

acquiring a data file of a first partition;

acquiring a data file of the newly-built partition; linking the data file of the newly-built partition with the data file of the first partition so as to create a link relation between the newly-built partition and the first partition;

before the step of linking the data file of the newly created partition with the data file of the first partition, the method comprises the following steps:

judging whether the data file in the first partition is a mapping link file;

2. The partition splitting method according to claim 1, wherein the data file comprises: physical files and map link files.

3. The partition splitting method of claim 2, wherein the linking the data files of the new partition with the data files of the first partition comprises:

4. The partition splitting method according to claim 3, wherein the creating a link relationship between the new partition and the first partition comprises:

5. The partition splitting method according to claim 4, wherein said step of creating a set of mapping links in said newly created partition is followed by:

and executing a persistence operation on the mapping link set.

6. The partition splitting method according to claim 4, wherein after the step of creating, for the new partition, a mapping link pointing to the first partition in which the data file pointed to by the mapping link file in the mapping link set is located, the method comprises:

receiving a partition scheduling request sent by the main service process;

7. The partition splitting method according to any one of claims 1 to 6, comprising, before the step of obtaining the data file of the first partition:

and creating at least two new partitions for the first partition.

8. The method of claim 7, wherein the creating at least two new partitions for the first partition comprises:

9. The partition splitting method according to claim 7, wherein after the step of linking the data files of the new partition with the data files of the first partition to create the linking relationship between the new partition and the first partition, the method comprises:

10. An apparatus for zone splitting, comprising:

a new data file obtaining unit, configured to obtain a data file of a new partition; a link relationship creating unit, configured to link the data file of the new partition with the data file of the first partition, so as to create a link relationship between the new partition and the first partition;

11. The partition splitting apparatus according to claim 10, wherein the data file obtaining unit is specifically configured to obtain a physical file and a mapping link file of the first partition.

12. The partition splitting apparatus according to claim 11, wherein the link relationship creating unit is specifically configured to point the mapping link file in the newly created partition to the data file in the first partition.

13. The apparatus for partition splitting according to claim 10, wherein the link relation creating unit further comprises:

14. The apparatus for partition splitting according to claim 13, wherein the link relation creating unit further comprises:

15. The zone splitting apparatus of claim 13, further comprising:

16. The zone splitting apparatus of any of claims 10-15, further comprising:

17. The apparatus for partition splitting according to claim 16, wherein the partition creating unit comprises:

18. The zone splitting apparatus of claim 16, further comprising:

19. A method for deleting a data file in a partition, comprising:

receiving an instruction for deleting a data file;

judging the type of the data file;

if the type of the data file is a mapping link file, deleting the data file from a mapping link set of a partition storing the data file;

after the step of deleting the data file in the mapping link set of the partition storing the data file, the method comprises the following steps:

20. The method of claim 19, wherein after the step of deleting the mapping link of the partition where the data file is deleted pointing to the partition where the previous data file pointed to by the deleted data file is located, the method comprises:

and if not, deleting the data files stored in the partitions.

21. An apparatus for deleting a data file in a partition, comprising:

the type judging unit is used for judging the type of the data file;

a mapping link file deleting unit, configured to receive a determination result of the type determining unit, and delete the data file in a mapping link set of a partition in which the data file is stored if the type of the data file is a mapping link file;

22. The apparatus for deleting a data file in a partition according to claim 21, further comprising:

23. A partition management system, comprising:

the apparatus of partition splitting according to any of the preceding claims 10 to 18; and

apparatus for deleting data files in a partition according to any of claims 21 to 22.