CN111026584A - Device, method and equipment for solving key value database file damage - Google Patents
Device, method and equipment for solving key value database file damage Download PDFInfo
- Publication number
- CN111026584A CN111026584A CN201911334407.7A CN201911334407A CN111026584A CN 111026584 A CN111026584 A CN 111026584A CN 201911334407 A CN201911334407 A CN 201911334407A CN 111026584 A CN111026584 A CN 111026584A
- Authority
- CN
- China
- Prior art keywords
- data
- value database
- key value
- key
- target storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1474—Saving, restoring, recovering or retrying in transactions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1469—Backup restoration techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/80—Database-specific techniques
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a device, a method and equipment for solving the problem of key value database file damage, comprising the following steps: acquiring data of a key value database; mapping virtual storage in local storage, and establishing volume mapping between the virtual storage and target storage; forwarding data of the key-value database to the virtual storage and the target storage; responding to a data writing request of the key value database, writing data in the key value database into virtual storage, and writing the data of the key value database into target storage according to volume mapping; and if the certain data is failed to be written into the target storage, forwarding the certain data written into the virtual storage to the target storage, and rewriting the certain data into the target storage. The method can not automatically skip the detection of the AOF file format, can not cause the loss of the data of the key value database, and reduces the risk of data loss; meanwhile, the method and the device do not need to adopt tools to repair the AOF file, greatly improve the maintenance efficiency and can be applied to wider application scenes.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a device, a method and equipment for solving key value database file damage.
Background
Redis is a high-performance key value database based on a memory, can respond to read-write requests ten thousand times per second, and is adopted by a large number of large and medium-sized websites at present. In order to ensure the complete durability of data, the Redis persists the data in an AOF (application Only File) mode, that is, stores a write command in a memory into a disk in an additional mode, and the method can ensure the safety of the data to the maximum extent. However, if the AOF persistence process is triggered, a server fails, which may cause damage to the AOF file, specifically, the write of the tail command of the AOF file is incomplete, thereby causing a Redis boot failure. The solution of the prior art is as follows: an AOF-load-truncated configuration is provided under which initiating Redis automatically skips AOF file format detection, however this approach may result in significant data loss. Alternatively, the AOF file is repaired using its own Redis-check-AOF tool before Redis starts, but the repair process takes longer when the AOF file is large.
Disclosure of Invention
In view of the above-mentioned shortcomings in the prior art, it is an object of the present invention to provide an apparatus, method and device for solving file corruption in a key-value database, so as to solve the technical problems in the prior art.
To achieve the above and other related objects, the present invention provides an apparatus for solving file corruption in a key-value database, comprising:
the acquisition module is used for acquiring data of the key value database;
the mapping module is used for mapping the virtual storage in the local storage and establishing volume mapping between the virtual storage and the target storage;
a forwarding module, configured to forward data of the key-value store to the virtual storage and the target storage;
the control module is used for responding to a data writing request of a key value database, writing data in the key value database into virtual storage, and writing the data of the key value database into the target storage according to the volume mapping; and if the certain data is failed to be written into the target storage, controlling the forwarding module to forward the certain data written in the virtual storage to the target storage, and rewriting the certain data into the target storage.
Optionally, the obtaining module is further configured to obtain data of a key value database cluster;
the forwarding module is further configured to forward data of the key-value database cluster to the virtual storage and the target storage;
the control module is further configured to respond to a data write request of a key-value database cluster, write data of the key-value database cluster into a virtual storage, and write data of the key-value database cluster into the target storage according to the volume mapping; and if the data of one key value database in the key value database cluster fails to be written into the target storage, controlling the forwarding module to forward the data of the other key value database in the key value database cluster written in the virtual storage to the target storage, and rewriting the data of the other key value database into the target storage.
Optionally, the control module further includes a detection unit, configured to detect whether a certain piece of data in the key-value store is successfully written into the target storage; and/or, the device is used for detecting whether data of a certain key value database in the key value database cluster is successfully written into the target storage.
Optionally, the control module comprises a clearing unit;
if a certain piece of data of the key value database is successfully written into a target storage, the clearing unit clears the certain piece of data written in the virtual storage;
and/or if the data of a certain key value database in the key value database cluster is successfully written into the target storage, the clearing unit clears the data of the certain key value database written into the virtual storage.
Optionally, the obtaining module further includes a configuration unit, and the configuration unit is configured to configure at least one of:
the key value database cluster account number, the key value database cluster password, the key value database cluster broadcasting domain, the target storage access mode, the mapping between the volume and the local storage and the write strategy of the data write request.
The invention also provides a method for solving the problem of key value database file damage, which comprises the following steps:
acquiring data of a key value database;
mapping virtual storage in local storage, and establishing volume mapping between the virtual storage and target storage;
forwarding data of the key-value store to the virtual storage and the target storage;
responding to a data writing request of a key-value database, writing data in the key-value database into virtual storage, and writing the data of the key-value database into the target storage according to the volume mapping; and if the certain data is not written into the target storage, forwarding the certain data written into the virtual storage to the target storage, and rewriting the certain data into the target storage.
Optionally, the method further includes: acquiring data of a key value database cluster;
forwarding data of the key-value database cluster to the virtual storage and the target storage;
responding to a data writing request of the key value database cluster, writing the data of the key value database cluster into a virtual storage, and writing the data of the key value database cluster into the target storage according to the volume mapping; if the data of one key value database in the key value database cluster fails to be written into the target storage, the data of the other key value database in the key value database cluster written in the virtual storage is forwarded to the target storage, and the data of the other key value database is rewritten into the target storage.
Optionally, detecting whether a piece of data in the key-value database is successfully written into the target storage; and/or detecting whether the data of one key value database in the key value database cluster is successfully written into the target storage.
Optionally, the method further includes: if a certain piece of data in the key value database is successfully written into a target storage, clearing the certain piece of data written in the virtual storage; and/or if the data of a certain key value database in the key value database cluster is successfully written into the target storage, clearing the data of the certain key value database written in the virtual storage.
The present invention also provides an apparatus comprising:
one or more processors; and
one or more machine-readable media having instructions stored thereon that, when executed by the one or more processors, cause the apparatus to perform a method as described in one or more of the above.
As described above, the present invention provides an apparatus, method and device for solving key value database file damage, which have the following beneficial effects: acquiring data of a key value database through an acquisition module; the mapping module maps virtual storage in local storage and establishes volume mapping between the virtual storage and target storage; a forwarding module forwards data of the key-value database to the virtual storage and the target storage; the control module responds to a data writing request of a key value database, writes data in the key value database into virtual storage, and writes the data of the key value database into the target storage according to the volume mapping; and if the certain data is failed to be written into the target storage, controlling the forwarding module to forward the certain data written in the virtual storage to the target storage, and rewriting the certain data into the target storage. The method maps the virtual storage in the local storage, establishes the volume mapping between the virtual storage and the target storage, synchronizes the data of the key value database to the target storage by establishing the volume mapping, and completes the persistence of the data, thereby ensuring the durability of the data. And according to the data writing request of the key value database, simultaneously writing the data into the virtual storage and the target storage, and if the writing into the target storage fails, forwarding the data written in the virtual storage to the target storage for rewriting. The method can not automatically skip the detection of the AOF file format, can not cause the data loss of the key value database, and reduces the risk of data loss; meanwhile, the method and the device do not need to adopt tools to repair the AOF file, greatly improve the maintenance efficiency and can be applied to wider application scenes.
Drawings
Fig. 1 is a connection diagram of an apparatus for solving file corruption in a key-value store according to an embodiment.
Fig. 2 is a flowchart illustrating a method for solving a problem of key-value database file corruption according to an embodiment.
Fig. 3 is a flowchart illustrating a method for solving key-value database file corruption according to another embodiment.
Fig. 4 is a schematic diagram of a hardware structure of a terminal device according to an embodiment.
Fig. 5 is a schematic diagram of a hardware structure of a terminal device according to another embodiment.
Description of the element reference numerals
M10 acquisition module
M20 mapping module
M30 forwarding module
M40 control module
1100 input device
1101 first processor
1102 output device
1103 first memory
1104 communication bus
1200 processing assembly
1201 second processor
1202 second memory
1203 communication assembly
1204 Power supply Assembly
1205 multimedia assembly
1206 voice assembly
1207 input/output interface
1208 sensor assembly
Detailed Description
The embodiments of the present invention are described below with reference to specific embodiments, and other advantages and effects of the present invention will be easily understood by those skilled in the art from the disclosure of the present specification. The invention is capable of other and different embodiments and of being practiced or of being carried out in various ways, and its several details are capable of modification in various respects, all without departing from the spirit and scope of the present invention. It is to be noted that the features in the following embodiments and examples may be combined with each other without conflict.
Please refer to fig. 1 to 5. It should be noted that the drawings provided in the present embodiment are only for illustrating the basic idea of the present invention, and the components related to the present invention are only shown in the drawings rather than drawn according to the number, shape and size of the components in actual implementation, and the type, quantity and proportion of the components in actual implementation may be changed freely, and the layout of the components may be more complicated. The structures, proportions, sizes, and other dimensions shown in the drawings and described in the specification are for understanding and reading the present disclosure, and are not intended to limit the scope of the present disclosure, which is defined in the claims, and are not essential to the art, and any structural modifications, changes in proportions, or adjustments in size, which do not affect the efficacy and attainment of the same are intended to fall within the scope of the present disclosure. In addition, the terms "upper", "lower", "left", "right", "middle" and "one" used in the present specification are for clarity of description, and are not intended to limit the scope of the present invention, and the relative relationship between the terms and the terms is not to be construed as a scope of the present invention.
Referring to fig. 1, the present embodiment provides an apparatus for solving a problem of a Redis file corruption in a key-value store, including:
an obtaining module M10, configured to obtain data of a key value database Redis;
the mapping module M20 is connected to the obtaining module M10, and is configured to map the virtual storage in the local storage, and establish a volume mapping between the virtual storage and the target storage. If the virtual storage comprises a volume file system of the key value database, directly mounting the target storage; if the volume file system is not included, the virtual storage is formatted first, and then the target storage is mounted. In the embodiment of the application, the data of the key value database is synchronized to the target storage by establishing the volume mapping, and the persistence of the data is completed, so that the durability of the data is ensured. In the embodiment of the application, the local storage is self-contained storage of the device, and the virtual storage is a virtual disk mapped according to the local storage; the target storage is a third-party shared storage, for example, including CEPH-RBD and ISCSI; wherein, RBD is a Reliable autonomous distributed Object storage Block Device (Reliable autonomous distributed Object storage Block Device), ISCSI is an Internet Small Computer System Interface (Internet Small Computer System Interface).
And the forwarding module M30 is connected to the mapping module M20, and is configured to forward data of the key-value database Redis to the virtual storage and the target storage. In this embodiment of the present application, the forwarding module M30 completes forwarding of data according to a preset forwarding rule. In practical situations, the corresponding forwarding rules can be selected according to requirements, the forwarding rules mainly include direct transparent transmission and high-availability writing, the direct transparent transmission is used for ensuring high performance, and the high-availability writing is used for ensuring high reliability.
And the control module M40 is connected to the forwarding module M30 and is used for responding to a data write request of the key-value database Redis. After the key-value database Redis issued with a write request, the control module M40 completes splitting of a Virtual File System (VFS) for the data write request, and converts the split data write request into a specific open/close command. And writing the data in the key value database Redis into the virtual storage, and writing the data in the key value database Redis into the target storage according to the volume mapping. If the write of a certain piece of data into the target storage fails, the control forwarding module M30 forwards the piece of data written in the virtual storage to the target storage, and rewrites the certain piece of data into the target storage. Specifically, for example, if the writing of the ith data into the target storage fails, the control module M40 controls the forwarding module M30 to forward the ith data written in the virtual storage to the target storage, and to rewrite the ith data into the target storage.
The device maps the virtual storage in the local storage, establishes the volume mapping between the virtual storage and the target storage, synchronizes the data of the key value database to the target storage by establishing the volume mapping, and completes the persistence of the data, thereby ensuring the durability of the data. And according to a data writing request sent by the key-value database, writing the data of the key-value database into the virtual storage and the target storage simultaneously. And if the target storage fails to be written, forwarding the data written in the virtual storage to the target storage for rewriting. By the scheme of the invention, the device can not automatically skip the detection of the AOF file format, can not cause the loss of the data of the Redis of the key value database, and reduces the risk of data loss; meanwhile, the method and the device do not need to adopt tools to repair the AOF file, greatly improve the maintenance efficiency and can be applied to wider application scenes.
In an exemplary embodiment, the obtaining module M10 is further configured to obtain data of a key-value store Redis cluster; the obtaining module M10 further includes a configuration unit, where the configuration unit is configured to configure one or more of the following: the method comprises the steps of key value database Redis cluster account numbers, key value database Redis cluster passwords, key value database Redis cluster broadcast domains, target storage access modes, mapping of volumes and local storage, write strategies of data write requests, user name accounts and user name passwords.
The mapping module M20 is connected to the obtaining module M10, and is configured to map the virtual storage in the local storage, and establish a volume mapping between the virtual storage and the target storage. If the virtual storage comprises a volume file system of the key value database, directly mounting the target storage; if the volume file system is not included, the virtual storage is formatted first, and then the target storage is mounted. In the embodiment of the application, the data of the key value database is synchronized to the target storage by establishing the volume mapping, and the persistence of the data is completed, so that the durability of the data is ensured. In the embodiment of the application, the local storage is self-contained storage of the device, and the virtual storage is a virtual disk mapped according to the local storage; the target storage is a third-party shared storage, for example, including CEPH-RBD and ISCSI; wherein, RBD is a Reliable autonomous distributed Object storage Block Device (Reliable autonomous distributed Object storage Block Device), ISCSI is an Internet Small Computer System Interface (Internet Small Computer System Interface).
The forwarding module M30 is further configured to forward data of the key-value store Redis cluster to the virtual storage and the target storage; in the embodiment of the present application, the forwarding module M30 completes forwarding of data according to a preset forwarding rule, and in an actual situation, a corresponding forwarding rule may be selected according to a requirement. The forwarding rules mainly include direct transparent transmission and high-availability writing, the direct transparent transmission is used for ensuring high performance, and the high-availability writing is used for ensuring high reliability. If a high available write is selected, data of a certain key-value database in the key-value database cluster can be forwarded to other key-value databases in the key-value database cluster for caching through a network.
The control module M40 is further configured to respond to a data write request of the key-value database Redis cluster, where after the key-value database Redis cluster sends the write request, the control module M40 completes splitting of a Virtual File System (VFS) for the data write request, and converts the split data write request into a specific open/close command. Writing the data of the Redis cluster of the key value database into a virtual storage, and writing the data of the Redis cluster of the key value database into a target storage according to volume mapping; if the data of one key value database Redis in the key value database Redis cluster fails to be written into the target storage, the control forwarding module M30 forwards the data of another key value database Redis in the key value database Redis cluster written into the virtual storage to the target storage, and rewrites the data of another key value database Redis in the target storage. Specifically, for example, if the data of the k-th key value database fails to be written into the target storage, the control module M40 controls the forwarding module M30 to forward the data of the j-th key value database written in the virtual storage to the target storage, and to rewrite the data of the j-th key value database into the target storage. And k is not equal to j, and the kth key value database and the jth key value database belong to the same key value database cluster.
The device also maps the virtual storage in the local storage, establishes volume mapping between the virtual storage and the target storage, synchronizes data of all key value databases in the key value database cluster to the target storage by establishing the volume mapping, and completes persistence of the data, thereby ensuring the durability of the data. According to a data writing request sent by the key value database cluster, data of all key value databases in the key value database cluster are written into the virtual storage and the target storage at the same time, and if the data of one key value database is failed to be written into the target storage, the data of the other key value database Redis in the Redis cluster written into the virtual storage is forwarded to the target storage for rewriting. The device can not cause the data of the key value database Redis in the key value database cluster to be lost, and reduces the risk of data loss; meanwhile, the method and the device do not need to adopt tools to repair the AOF file, greatly improve the maintenance efficiency and can be applied to wider application scenes.
In some embodiments, the control module further includes a detecting unit, configured to detect whether a piece of data in the key-value store Redis successfully written into the target storage; and/or, the method is used for detecting whether data of a certain key value database Redis in the key value database Redis cluster is successfully written into the target storage. The control module comprises a clearing unit; if a certain piece of data in the key value database Redis is successfully written into the target storage, the clearing unit clears the certain piece of data written in the virtual storage; and/or if the data of a certain key value database Redis in the key value database Redis cluster is successfully written into the target storage, the clearing unit clears the data of the certain key value database Redis written in the virtual storage.
Specifically, if the detection unit detects that a certain piece of data in the key-value database Redis is successfully written into the target storage, the target storage sends a first write signal to the control module, and the control module controls the clearing unit to clear the piece of data written in the virtual storage and perform writing of the next piece of data. If the detection unit detects that a certain piece of data in the Redis is written into the target storage unsuccessfully, the target storage sends a first unwritten signal to the control module, and the control module controls the forwarding module to forward the piece of data written in the virtual storage to the target storage and rewrite the piece of data into the target storage.
If the detection unit detects that data of a certain key value database Redis in the key value database Redis cluster is successfully written into the target storage, the target storage sends a second write-in signal to the control module, and the control module controls the clearing unit to clear the data of the certain key value database Redis written in the virtual storage and write the data of the next key value database. Each key value database in the key value database cluster sends heartbeat information to each other, if the detection unit detects that more than half of the key value databases in the key value database Redis cluster do not receive heartbeat information of one key value database, the detection unit determines that the data of one key value database Redis in the key value database Redis cluster fails to be written into the target storage, the target storage sends a second unwritten signal to the control module, the control module controls the forwarding module to forward the data of the other key value database Redis in the key value database Redis cluster written in the virtual storage to the target storage, and the data of the other key value database Redis is rewritten into the target storage.
In summary, the apparatus maps the virtual storage in the local storage, establishes a volume mapping between the virtual storage and the target storage, synchronizes the data of the key-value database and/or the data of the key-value database in the key-value database cluster to the target storage by establishing the volume mapping, and completes the persistence of the data, thereby ensuring the durability of the data. And according to the sent data writing request, writing the data into the virtual storage and the target storage simultaneously. And if the writing of certain data into the target storage fails, forwarding the corresponding data written in the virtual storage to the target storage for rewriting. Through the scheme of the invention, the method for solving the problem of Redis persistent file format damage can ensure the integrity of the AOF file under the condition that the Redis service provides node failure. Compared with the prior art, the method has the advantages that the stability is enhanced, and the risk of data loss is effectively reduced; in addition, the invention also avoids adopting a redis-check-aof tool to repair, greatly improves the maintenance efficiency, and is suitable for wider application scenes.
As shown in fig. 2, the present invention further provides a method for solving the problem of Redis file corruption in a key-value database, the method comprising the following steps:
s100, acquiring data of a key value database Redis;
s200, mapping virtual storage in local storage, and establishing volume mapping between the virtual storage and target storage; if the virtual storage comprises a volume file system of the key value database, directly mounting the target storage; if the volume file system is not included, the virtual storage is formatted first, and then the target storage is mounted. In the embodiment of the application, the data of the key value database is synchronized to the target storage by establishing the volume mapping, and the persistence of the data is completed, so that the durability of the data is ensured. In the embodiment of the application, the local storage is self-contained storage of the device, and the virtual storage is a virtual disk mapped according to the local storage; the target storage is a third-party shared storage, for example, including CEPH-RBD and ISCSI; wherein, RBD is a Reliable autonomous Distributed Object storage block device (Reliable autonomous Distributed Object storage Block device), ISCSI is an Internet Small Computer System interface (Internet Small Computer System interface).
S300, forwarding data of the Redis key value database to a virtual storage and a target storage; in the embodiment of the application, the forwarding module completes the forwarding of the data according to the preset forwarding rule. In practical situations, the corresponding forwarding rules can be selected according to requirements, the forwarding rules mainly include direct transparent transmission and high-availability writing, the direct transparent transmission is used for ensuring high performance, and the high-availability writing is used for ensuring high reliability.
S400, responding to a data write request of the key value database Redis, controlling a module to complete the splitting of a Virtual File System (VFS) to the data write request after the key value database Redis sends the write request, and converting the split data write request into a specific open/close command. Writing the data in the key value database Redis into the virtual storage, and writing the data in the key value database Redis into a target storage according to volume mapping; and if the certain data is failed to be written into the target storage, forwarding the certain data written into the virtual storage to the target storage, and rewriting the certain data into the target storage. Specifically, for example, if writing of the ith piece of data into the target storage fails, the control module controls the forwarding module to forward the ith piece of data written in the virtual storage to the target storage, and writes the ith piece of data into the target storage again.
The method maps the virtual storage in the local storage, establishes the volume mapping between the virtual storage and the target storage, synchronizes the data of the key value database to the target storage by establishing the volume mapping, and completes the persistence of the data, thereby ensuring the durability of the data. And according to a data writing request sent by the key-value database, writing the data of the key-value database into the virtual storage and the target storage simultaneously. And if the target storage fails to be written, forwarding the data written in the virtual storage to the target storage for rewriting. By the scheme, the detection of the AOF file format cannot be automatically skipped, the data loss of the Redis of the key value database cannot be caused, and the risk of data loss is reduced; meanwhile, the method and the device do not need to adopt tools to repair the AOF file, greatly improve the maintenance efficiency and can be applied to wider application scenes.
As shown in fig. 3, in an exemplary embodiment, the method for solving the corruption of the Redis file in the key-value store further includes:
s500, acquiring data of a Redis cluster of a key value database;
s600, mapping virtual storage in local storage, and establishing volume mapping between the virtual storage and target storage; if the virtual storage comprises a volume file system of the key value database, directly mounting the target storage; if the volume file system is not included, the virtual storage is formatted first, and then the target storage is mounted. In the embodiment of the application, the data of the key value database is synchronized to the target storage by establishing the volume mapping, and the persistence of the data is completed, so that the durability of the data is ensured. In the embodiment of the application, the local storage is self-contained storage of the device, and the virtual storage is a virtual disk mapped according to the local storage; the target storage is a third-party shared storage, for example, including CEPH-RBD and ISCSI; wherein, RBD is a Reliable autonomous Distributed Object storage block device (Reliable autonomous Distributed Object storage Block device), ISCSI is an Internet Small Computer System interface (Internet Small Computer System interface).
S700, forwarding data of the Redis cluster of the key value database to virtual storage and target storage; in the embodiment of the application, the forwarding module completes the forwarding of the data according to the preset forwarding rule, and the corresponding forwarding rule can be selected according to the requirement in an actual situation. The forwarding rules mainly include direct transparent transmission and high-availability writing, the direct transparent transmission is used for ensuring high performance, and the high-availability writing is used for ensuring high reliability. If a high available write is selected, data of a certain key-value database in the key-value database cluster can be forwarded to other key-value databases in the key-value database cluster for caching through a network.
S800, responding to a data write request of the Redis cluster of the key-value database, after the Redis cluster of the key-value database sends the write request, controlling a module to complete the splitting of a Virtual File System (VFS) to the data write request, and converting the split data write request into a specific open/close command. Writing data of the Redis cluster of the key value database into virtual storage, and writing the data of the Redis cluster of the key value database into target storage according to volume mapping; if the data of one key value database Redis in the key value database Redis cluster fails to be written into the target storage, the data of the other key value database Redis in the key value database Redis cluster written in the virtual storage is forwarded to the target storage, and the data of the other key value database Redis is rewritten into the target storage. Specifically, for example, if the data of the k-th key value database fails to be written into the target storage, the control module controls the forwarding module to forward the data of the j-th key value database written in the virtual storage to the target storage, and to write the data of the j-th key value database into the target storage again. And k is not equal to j, and the kth key value database and the jth key value database belong to the same key value database cluster.
The method also maps the virtual storage in the local storage, establishes volume mapping between the virtual storage and the target storage, synchronizes data of all key value databases in the key value database cluster to the target storage by establishing the volume mapping, and completes persistence of the data, thereby ensuring the durability of the data. According to a data writing request sent by the key value database cluster, data of all key value databases in the key value database cluster are written into the virtual storage and the target storage at the same time, and if the data of one key value database is failed to be written into the target storage, the data of the other key value database Redis in the Redis cluster written into the virtual storage is forwarded to the target storage for rewriting. The method can not cause the data of the key value database Redis in the key value database cluster to be lost, and reduces the risk of data loss; meanwhile, the method and the device do not need to adopt tools to repair the AOF file, greatly improve the maintenance efficiency and can be applied to wider application scenes.
In some embodiments, the method further comprises detecting whether a piece of data in the key-value store Redis is successfully written into the target storage; and/or detecting whether data of a certain key value database Redis in the Redis cluster of the key value database is successfully written into the target storage. Specifically, the method further comprises the following steps: if a certain piece of data in the key value database Redis is successfully written into the target storage, clearing the certain piece of data written in the virtual storage; and/or if the data of a certain key value database Redis in the key value database Redis cluster is successfully written into the target storage, clearing the data of the certain key value database Redis written in the virtual storage.
Specifically, if the detection unit detects that a certain piece of data in the key-value database Redis is successfully written into the target storage, the target storage sends a first write signal to the control module, and the control module controls the clearing unit to clear the piece of data written in the virtual storage and perform writing of the next piece of data. If the detection unit detects that a certain piece of data in the Redis is written into the target storage unsuccessfully, the target storage sends a first unwritten signal to the control module, and the control module controls the forwarding module to forward the piece of data written in the virtual storage to the target storage and rewrite the piece of data into the target storage.
If the detection unit detects that data of a certain key value database Redis in the key value database Redis cluster is successfully written into the target storage, the target storage sends a second write-in signal to the control module, and the control module controls the clearing unit to clear the data of the certain key value database Redis written in the virtual storage and write the data of the next key value database. Each key value database in the key value database cluster sends heartbeat information to each other, if the detection unit detects that more than half of the key value databases in the key value database Redis cluster do not receive heartbeat information of one key value database, the detection unit determines that the data of one key value database Redis in the key value database Redis cluster fails to be written into the target storage, the target storage sends a second unwritten signal to the control module, the control module controls the forwarding module to forward the data of the other key value database Redis in the key value database Redis cluster written in the virtual storage to the target storage, and the data of the other key value database Redis is rewritten into the target storage.
In summary, the method maps the virtual storage in the local storage, establishes the volume mapping between the virtual storage and the target storage, synchronizes the data of the key value database and/or the data of the key value database in the key value database cluster to the target storage by establishing the volume mapping, and completes the persistence of the data, thereby ensuring the durability of the data. And according to the sent data writing request, writing the data into the virtual storage and the target storage simultaneously. And if the writing of certain data into the target storage fails, forwarding the corresponding data written in the virtual storage to the target storage for rewriting. By the scheme, the method for solving the problem of the Redis persistent file format damage can ensure the integrity of the AOF file under the condition that the Redis service providing node fails. Compared with the prior art, the method has the advantages that the stability is enhanced, and the risk of data loss is effectively reduced; in addition, the invention also avoids adopting a redis-check-aof tool to repair, greatly improves the maintenance efficiency, and is suitable for wider application scenes.
An embodiment of the present application further provides an apparatus, which may include: one or more processors; and one or more machine readable media having instructions stored thereon that, when executed by the one or more processors, cause the apparatus to perform the methods of fig. 2 and 3. In practical applications, the device may be used as a terminal device, and may also be used as a server, where examples of the terminal device may include: the mobile terminal includes a smart phone, a tablet computer, an electronic book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop, a vehicle-mounted computer, a desktop computer, a set-top box, an intelligent television, a wearable device, and the like.
The present application further provides a non-transitory readable storage medium, where one or more modules (programs) are stored in the storage medium, and when the one or more modules are applied to a device, the device may execute instructions (instructions) of steps included in the methods in fig. 2 and fig. 3 according to the present application.
Fig. 4 is a schematic diagram of a hardware structure of a terminal device according to an embodiment of the present application. As shown, the terminal device may include: an input device 1100, a first processor 1101, an output device 1102, a first memory 1103, and at least one communication bus 1104. The communication bus 1104 is used to implement communication connections between the elements. The first memory 1103 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and the first memory 1103 may store various programs for performing various processing functions and implementing the method steps of the present embodiment.
Alternatively, the first processor 1101 may be, for example, a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic components, and the first processor 1101 is coupled to the input device 1100 and the output device 1102 through a wired or wireless connection.
Optionally, the input device 1100 may include a variety of input devices, such as at least one of a user-oriented user interface, a device-oriented device interface, a software programmable interface, a camera, and a sensor. Optionally, the device interface facing the device may be a wired interface for data transmission between devices, or may be a hardware plug-in interface (e.g., a USB interface, a serial port, etc.) for data transmission between devices; optionally, the user-facing user interface may be, for example, a user-facing control key, a voice input device for receiving voice input, and a touch sensing device (e.g., a touch screen with a touch sensing function, a touch pad, etc.) for receiving user touch input; optionally, the programmable interface of the software may be, for example, an entry for a user to edit or modify a program, such as an input pin interface or an input interface of a chip; the output devices 1102 may include output devices such as a display, audio, and the like.
In this embodiment, the processor of the terminal device includes a function for executing each module of the speech recognition apparatus in each device, and specific functions and technical effects may refer to the above embodiments, which are not described herein again.
Fig. 5 is a schematic hardware structure diagram of a terminal device according to an embodiment of the present application. Fig. 5 is a specific embodiment of the implementation process of fig. 4. As shown, the terminal device of the present embodiment may include a second processor 1201 and a second memory 1202.
The second processor 1201 executes the computer program code stored in the second memory 1202 to implement the methods described in fig. 2 and 3 in the above embodiments.
The second memory 1202 is configured to store various types of data to support operations at the terminal device. Examples of such data include instructions for any application or method operating on the terminal device, such as messages, pictures, videos, and so forth. The second memory 1202 may include a Random Access Memory (RAM) and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
Optionally, a second processor 1201 is provided in the processing assembly 1200. The terminal device may further include: communication component 1203, power component 1204, multimedia component 1205, speech component 1206, input/output interfaces 1207, and/or sensor component 1208. The specific components included in the terminal device are set according to actual requirements, which is not limited in this embodiment.
The processing component 1200 generally controls the overall operation of the terminal device. The processing assembly 1200 may include one or more second processors 1201 to execute instructions to perform all or part of the steps of the data processing method described above. Further, the processing component 1200 can include one or more modules that facilitate interaction between the processing component 1200 and other components. For example, the processing component 1200 can include a multimedia module to facilitate interaction between the multimedia component 1205 and the processing component 1200.
The power supply component 1204 provides power to the various components of the terminal device. The power components 1204 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the terminal device.
The multimedia components 1205 include a display screen that provides an output interface between the terminal device and the user. In some embodiments, the display screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the display screen includes a touch panel, the display screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.
The voice component 1206 is configured to output and/or input voice signals. For example, the voice component 1206 includes a Microphone (MIC) configured to receive external voice signals when the terminal device is in an operational mode, such as a voice recognition mode. The received speech signal may further be stored in the second memory 1202 or transmitted via the communication component 1203. In some embodiments, the speech component 1206 further comprises a speaker for outputting speech signals.
The input/output interface 1207 provides an interface between the processing component 1200 and peripheral interface modules, which may be click wheels, buttons, etc. These buttons may include, but are not limited to: a volume button, a start button, and a lock button.
The sensor component 1208 includes one or more sensors for providing various aspects of status assessment for the terminal device. For example, the sensor component 1208 may detect an open/closed state of the terminal device, relative positioning of the components, presence or absence of user contact with the terminal device. The sensor assembly 1208 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact, including detecting the distance between the user and the terminal device. In some embodiments, the sensor assembly 1208 may also include a camera or the like.
The communication component 1203 is configured to facilitate communications between the terminal device and other devices in a wired or wireless manner. The terminal device may access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In one embodiment, the terminal device may include a SIM card slot therein for inserting a SIM card therein, so that the terminal device may log onto a GPRS network to establish communication with the server via the internet.
As can be seen from the above, the communication component 1203, the voice component 1206, the input/output interface 1207 and the sensor component 1208 involved in the embodiment of fig. 5 can be implemented as the input device in the embodiment of fig. 4.
The foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical spirit of the present invention be covered by the claims of the present invention.
Claims (10)
1. An apparatus for solving file corruption in a key-value store, comprising:
the acquisition module is used for acquiring data of the key value database;
the mapping module is used for mapping the virtual storage in the local storage and establishing volume mapping between the virtual storage and the target storage;
a forwarding module, configured to forward data of the key-value store to the virtual storage and the target storage;
the control module is used for responding to a data writing request of a key value database, writing data in the key value database into virtual storage, and writing the data of the key value database into the target storage according to the volume mapping; and if the certain data is failed to be written into the target storage, controlling the forwarding module to forward the certain data written in the virtual storage to the target storage, and rewriting the certain data into the target storage.
2. The apparatus of resolving key-value database file corruptions of claim 1, wherein: the obtaining module is further used for obtaining data of the key value database cluster;
the forwarding module is further configured to forward data of the key-value database cluster to the virtual storage and the target storage;
the control module is further configured to respond to a data write request of a key-value database cluster, write data of the key-value database cluster into a virtual storage, and write data of the key-value database cluster into the target storage according to the volume mapping; and if the data of one key value database in the key value database cluster fails to be written into the target storage, controlling the forwarding module to forward the data of the other key value database in the key value database cluster written in the virtual storage to the target storage, and rewriting the data of the other key value database into the target storage.
3. The apparatus of resolving key-value database file corruptions of claim 2, wherein: the control module comprises a detection unit for detecting whether a certain piece of data in the key value database is successfully written into the target storage; and/or, the device is used for detecting whether data of a certain key value database in the key value database cluster is successfully written into the target storage.
4. The apparatus for resolving file corruption in a key-value database as recited in claim 2 or claim 3, wherein: the control module also comprises a clearing unit; if a certain piece of data of the key value database is successfully written into a target storage, the clearing unit clears the certain piece of data written in the virtual storage;
and/or if the data of a certain key value database in the key value database cluster is successfully written into the target storage, the clearing unit clears the data of the certain key value database written into the virtual storage.
5. The apparatus of resolving key-value database file corruptions of claim 3, wherein: the acquisition module comprises a configuration unit, and the configuration unit is used for configuring at least one of the following:
the key value database cluster account number, the key value database cluster password, the key value database cluster broadcasting domain, the target storage access mode, the mapping between the volume and the local storage and the write strategy of the data write request.
6. A method for resolving key-value database file corruption, the method comprising the steps of:
acquiring data of a key value database;
mapping virtual storage in local storage, and establishing volume mapping between the virtual storage and target storage;
forwarding data of the key-value store to the virtual storage and the target storage;
responding to a data writing request of a key-value database, writing data in the key-value database into virtual storage, and writing the data of the key-value database into the target storage according to the volume mapping; and if the certain data is not written into the target storage, forwarding the certain data written into the virtual storage to the target storage, and rewriting the certain data into the target storage.
7. The method of resolving key-value database file corruption of claim 6, wherein: also includes: acquiring data of a key value database cluster;
forwarding data of the key-value database cluster to the virtual storage and the target storage;
responding to a data writing request of the key value database cluster, writing the data of the key value database cluster into a virtual storage, and writing the data of the key value database cluster into the target storage according to the volume mapping; if the data of one key value database in the key value database cluster fails to be written into the target storage, the data of the other key value database in the key value database cluster written in the virtual storage is forwarded to the target storage, and the data of the other key value database is rewritten into the target storage.
8. The method of resolving key-value database file corruption of claim 7, wherein: detecting whether certain data in the key value database is successfully written into the target storage; and/or detecting whether the data of one key value database in the key value database cluster is successfully written into the target storage.
9. The method of resolving key-value database file corruption of claim 7, wherein: also includes: if a certain piece of data in the key value database is successfully written into a target storage, clearing the certain piece of data written in the virtual storage; and/or if the data of a certain key value database in the key value database cluster is successfully written into the target storage, clearing the data of the certain key value database written in the virtual storage.
10. An apparatus, comprising:
one or more processors; and
one or more machine-readable media having instructions stored thereon that, when executed by the one or more processors, cause the apparatus to perform the method recited by one or more of claims 6-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911334407.7A CN111026584A (en) | 2019-12-23 | 2019-12-23 | Device, method and equipment for solving key value database file damage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911334407.7A CN111026584A (en) | 2019-12-23 | 2019-12-23 | Device, method and equipment for solving key value database file damage |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111026584A true CN111026584A (en) | 2020-04-17 |
Family
ID=70212551
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911334407.7A Pending CN111026584A (en) | 2019-12-23 | 2019-12-23 | Device, method and equipment for solving key value database file damage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111026584A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112817973A (en) * | 2021-01-22 | 2021-05-18 | 平安普惠企业管理有限公司 | Data processing method, data processing device, data processing equipment and storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102375892A (en) * | 2011-11-16 | 2012-03-14 | 北京握奇数据***有限公司 | Large-capacity database card and office management method thereof, and system |
| CN104794155A (en) * | 2015-03-12 | 2015-07-22 | 北京奇虎科技有限公司 | Data loading method, device and system |
| CN105471714A (en) * | 2015-12-09 | 2016-04-06 | 百度在线网络技术（北京）有限公司 | Message processing method and device |
| CN105718592A (en) * | 2016-01-27 | 2016-06-29 | 北京铭万智达科技有限公司 | Data calling method and system based on Redis |
| CN105739924A (en) * | 2016-01-29 | 2016-07-06 | 华为技术有限公司 | Cache cluster-based cache method and system |
| CN107122364A (en) * | 2016-02-25 | 2017-09-01 | 华为技术有限公司 | Data manipulation method and data management server |
| CN107329858A (en) * | 2017-07-10 | 2017-11-07 | 上海哎媲媲网络技术有限公司 | A kind of Quick fault-tolerance processing method of data write-in Redis failures |
| US10235375B1 (en) * | 2011-05-14 | 2019-03-19 | Pivotal Software, Inc. | Persistent file system objects for management of databases |
| CN109614270A (en) * | 2018-12-14 | 2019-04-12 | 深圳前海微众银行股份有限公司 | Data read-write method, device, equipment and storage medium based on Hbase |
-
2019
- 2019-12-23 CN CN201911334407.7A patent/CN111026584A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10235375B1 (en) * | 2011-05-14 | 2019-03-19 | Pivotal Software, Inc. | Persistent file system objects for management of databases |
| CN102375892A (en) * | 2011-11-16 | 2012-03-14 | 北京握奇数据***有限公司 | Large-capacity database card and office management method thereof, and system |
| CN104794155A (en) * | 2015-03-12 | 2015-07-22 | 北京奇虎科技有限公司 | Data loading method, device and system |
| CN105471714A (en) * | 2015-12-09 | 2016-04-06 | 百度在线网络技术（北京）有限公司 | Message processing method and device |
| CN105718592A (en) * | 2016-01-27 | 2016-06-29 | 北京铭万智达科技有限公司 | Data calling method and system based on Redis |
| CN105739924A (en) * | 2016-01-29 | 2016-07-06 | 华为技术有限公司 | Cache cluster-based cache method and system |
| CN107122364A (en) * | 2016-02-25 | 2017-09-01 | 华为技术有限公司 | Data manipulation method and data management server |
| CN107329858A (en) * | 2017-07-10 | 2017-11-07 | 上海哎媲媲网络技术有限公司 | A kind of Quick fault-tolerance processing method of data write-in Redis failures |
| CN109614270A (en) * | 2018-12-14 | 2019-04-12 | 深圳前海微众银行股份有限公司 | Data read-write method, device, equipment and storage medium based on Hbase |
Non-Patent Citations (2)
| Title |
|---|
| 周立功 等: "《面向AWorks框架和接口的C编程（上）》", 30 November 2018, 北京航空航天大学出版社 * |
| 韩彦岭 等: "《计算机操作***》", 31 January 2018, 上海科学技术出版社 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112817973A (en) * | 2021-01-22 | 2021-05-18 | 平安普惠企业管理有限公司 | Data processing method, data processing device, data processing equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107273083B (en) | Interaction method, device, equipment and storage medium between terminal equipment | |
| WO2017028514A1 (en) | Method and device for storing and reading data | |
| KR102336528B1 (en) | Electronic device having cache memory and method for operating thereof | |
| US10761730B2 (en) | Method for configuring disk array of electronic device and related electronic device | |
| US10489289B1 (en) | Physical media aware spacially coupled journaling and trim | |
| CN114925084B (en) | Distributed transaction processing method, system, equipment and readable storage medium | |
| US20180067890A1 (en) | Embedding protocol parameters in data streams between host devices and storage devices | |
| CN107678693A (en) | Hanging method, Android device and the storage medium of disk array | |
| CN114579055B (en) | Disk storage method, device, equipment and medium | |
| CN111026584A (en) | Device, method and equipment for solving key value database file damage | |
| US20230367515A1 (en) | Storing and recovering critical data in a memory device | |
| CN116719724B (en) | EMMC multi-platform testing method and device and electronic equipment | |
| CN110515662B (en) | Remote control method, system, equipment and readable storage medium for dual-system equipment | |
| CN107728945A (en) | Hanging method, Android device and the storage medium of disk array | |
| CN104598453A (en) | Data migration method based on data buffering | |
| CN107609119B (en) | File processing method, mobile terminal and computer readable storage medium | |
| CN108369555B (en) | Logical address history management in a memory device | |
| CN113272785A (en) | Method for mounting file system, terminal equipment and storage medium | |
| CN111935405B (en) | Camera equipment control method, system, equipment and medium | |
| CN114070892A (en) | Data transmission method and device | |
| CN111352357B (en) | Robot control method and device and terminal equipment | |
| US20190294806A1 (en) | Method and system for implementing data secure deletion in mobile terminal | |
| KR20210101693A (en) | Electronic device including storage and method for issuing discard command of file system to storage using the same | |
| CN111243254A (en) | Wireless transmission circuit, device and equipment | |
| CN111625184A (en) | Flash memory data burning method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200417 |
|
| RJ01 | Rejection of invention patent application after publication |