CN110413237B - Distributed storage method, device, terminal equipment and storage medium - Google Patents
Distributed storage method, device, terminal equipment and storage medium Download PDFInfo
- Publication number
- CN110413237B CN110413237B CN201910730894.2A CN201910730894A CN110413237B CN 110413237 B CN110413237 B CN 110413237B CN 201910730894 A CN201910730894 A CN 201910730894A CN 110413237 B CN110413237 B CN 110413237B
- Authority
- CN
- China
- Prior art keywords
- storage
- document
- storage server
- server
- path
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Databases & Information Systems (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a distributed storage method, a device, a terminal device and a storage medium, wherein the method comprises the following steps: identifying a storage space occupied by a document to be stored; acquiring performance indexes of storage nodes of each storage server, matching storage space with the performance indexes, and determining a first storage server and a first storage path, and a second storage server and a second storage path; storing the document into a first storage server as a main document, and compressing and storing the document into a second storage server as an auxiliary document; if the first storage server fails, decompressing the auxiliary document to obtain a main document; and determining a third storage server and a third storage path according to the storage space occupied by the main document and a preset storage threshold, compressing the main document and storing the main document to the third storage server according to the third storage path. In the process of storing the document, the aims of automatically selecting the optimal storage path and ensuring the document storage safety are fulfilled.
Description
Technical Field
The invention relates to the technical field of document storage in a local area network, in particular to a distributed storage method, a distributed storage device, terminal equipment and a storage medium.
Background
With the rapid development of the current information age, the internet has occupied an important position in people's daily life and work, and people usually upload some document data to a mobile phone, a computer or other terminal devices to store the documents no matter in life or work. Especially for company enterprises, a large amount of document data needs to be stored, and once the problem of document loss occurs, especially for valuable documents with higher value, huge loss is caused, so that the security and reliability of document storage are guaranteed, which is a crucial problem for both individuals and enterprises.
In the prior art, as for a document storage method, documents are generally stored on a single storage device, and to ensure the security of document storage, multiple copies of documents are usually stored at different positions of a device, so that a user can freely access any copy in a device system. However, because the data volume of the document is large and the number of copies of the document is large, the storage space of the device is easily insufficient, so that the number of the documents actually stored is small. In addition, because of the difference in the performance of the storage devices, there is a great difference in storage efficiency and security, for example, when a document is stored on some storage devices with a small memory, a low processing speed of a central processing unit, or a low success rate of storage, not only the storage efficiency is reduced, but also the security of document storage is threatened. Therefore, the method for storing documents in the prior art cannot timely and effectively ensure the security of the documents when the storage device fails.
Disclosure of Invention
In view of this, a distributed storage method, an apparatus, a terminal device and a storage medium are provided to solve the problems in the prior art that the document storage space is insufficient, the storage efficiency is low and the document storage security cannot be guaranteed.
In order to achieve the purpose, the invention adopts the following technical scheme:
in a first aspect, a distributed storage method is used, and the method includes: identifying a storage space occupied by a document to be stored; the method comprises the steps of obtaining performance indexes of storage nodes of each storage server, matching storage space with the performance indexes, and determining a first storage server and a second storage server, as well as a first storage path corresponding to the first storage server and a second storage path corresponding to the second storage server, wherein the number of the first storage server and the second storage server is at least one; storing the document into a first storage server according to a first storage path to be used as a main document; compressing the document and storing the document into a second storage server according to a second storage path to be used as a side document;
if the first storage server fails, decompressing the secondary document to obtain a primary document; and determining a third storage server and a third storage path corresponding to the third storage server according to the storage space occupied by the main document and a preset storage threshold, compressing the main document and storing the main document to the third storage server according to the third storage path, wherein the preset storage threshold is determined by the total number of the storage servers in the current local area network and the performance indexes of the storage servers.
In a second aspect, a distributed storage apparatus is employed, the apparatus comprising: the identification module is used for identifying the storage space occupied by the document to be stored; the determining module is used for acquiring the performance indexes of the storage nodes of each storage server, matching the storage space with the performance indexes, and determining a first storage server, a second storage server, a first storage path corresponding to the first storage server and a second storage path corresponding to the second storage server, wherein the number of the first storage server and the second storage server is at least one; the first storage module is used for storing the document into a first storage server according to a first storage path to be used as a main document; the second storage module is used for compressing the document and storing the document into a second storage server as a secondary document according to a second storage path; the control module is used for decompressing the auxiliary document to obtain a main document if the first storage server fails; and determining a third storage server and a third storage path corresponding to the third storage server according to the storage space occupied by the main document and a preset storage threshold, compressing the main document and storing the main document to the third storage server according to the third storage path, wherein the preset storage threshold is determined by the total number of the storage servers in the current local area network and the performance indexes of the storage servers.
In a third aspect, a terminal device is adopted, where the terminal device includes: a processor, and a memory coupled to the processor; the memory is used for storing a computer program used for executing the distributed storage method; the processor is used to call and execute the computer program in the memory.
In a fourth aspect, a storage medium is used, which stores a computer program that, when executed by a processor, implements the steps of the above-described distributed storage method.
According to the technical scheme, firstly, the storage space occupied by the document is identified according to the document uploaded by a user; selecting a storage server which accords with the storage document according to the performance indexes of the storage server nodes and the storage space and determining the storage path of the document in the storage server, and selecting the optimal storage path for the document so as to improve the storage efficiency of the document; storing the documents according to a storage path, so that a user can conveniently and quickly find the storage position of the documents, wherein the main document is stored in a first storage server, and the auxiliary document is stored in a second storage server; and if the first storage server fails, decompressing the secondary document to become the primary document, reselecting a proper third storage server again, and compressing the primary document to serve as a new secondary document so as to ensure the safety and reliability of document storage. Through the technical scheme, in the document storage process, a proper storage path can be selected for document storage, the storage efficiency is improved, and a safe and reliable storage service is provided for a user.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart of a distributed storage method according to an embodiment of the present invention;
FIG. 2 is a flow chart of a distributed storage method according to an embodiment of the present invention;
FIG. 3 is a flow chart of a distributed storage method according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a distributed storage apparatus according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a terminal device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be described in detail below. It should be apparent that the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the examples given herein without any inventive step, are within the scope of the present invention.
Examples
Fig. 1 is a flowchart of a distributed storage method according to an embodiment of the present invention, where the method may be executed by a distributed storage apparatus according to an embodiment of the present invention, and the apparatus may be implemented in software and/or hardware. Referring to fig. 1, the method may specifically include the following steps:
s101, identifying a storage space occupied by a document to be stored.
Specifically, after determining the document to be stored, the processor of the storage server calculates the storage space required by the document to be stored according to the computer programming for calculating the storage capacity in the prior art and a software algorithm, so as to select a suitable storage server for the document to be stored according to the size of the storage space required by the document to be stored. For example, if a user needs to store a 1GB (GigaByte) document, but the remaining storage space of the storage server is only 0.5GB, the storage server with insufficient storage space is automatically ignored during the document storage process.
S102, obtaining performance indexes of storage nodes of each storage server, matching storage space with the performance indexes, and determining a first storage server and a second storage server, as well as a first storage path corresponding to the first storage server and a second storage path corresponding to the second storage server, wherein the number of the first storage server and the second storage server is at least one.
Specifically, in this embodiment, when the document is stored in the current lan, there are a plurality of storage servers, and the performance index of each storage server is different, so that the performance index of each storage server needs to be compared with the storage space occupied by the document to be stored, so as to select a storage server suitable for storing the document. In the practical application process, the registration monitoring center belongs to a part of the processor, the registration monitoring center acquires the performance indexes of each storage server, the processor matches the performance indexes with the storage space of the document to determine a first storage server and a second storage server for storing the document, and in addition, because each storage server is internally provided with a plurality of storage spaces, the storage server can be determined, and meanwhile, the storage path in each storage server can be determined, namely, when the first storage server and the second storage server for storing the document are determined, a first storage path corresponding to the first storage server and a second storage path corresponding to the second storage server are also determined, so that when a user searches for the document, the storage position where the document is can be quickly found according to the storage paths.
For example, the size of the storage space occupied by the document to be stored is 1GB, 5 storage servers are in total in the current local area network, and the storage amounts of the 5 storage servers are respectively: 0.5GB, 0.75GB, 1GB, 1.5GB and 3GB, wherein the storage capacity of the storage servers of 0.5GB and 0.75GB is smaller than the storage space of the document to be stored, therefore, the document can not be stored, two storage servers are selected from the remaining three storage servers, and the storage space of the document to be stored is 1GB, therefore, in order to better store the document, the storage capacities of 1.5GB and 3GB are selected as the first storage server and the second storage server.
In addition, in the process of storing the document, whether the document is stored in blocks can be determined according to the size of the document. Comparing the storage space occupied by the stored document with a preset storage space threshold, and if the storage space occupied by the document is smaller than the preset storage space threshold, selecting a proper storage server to directly store the document according to the scheme; if the storage space occupied by the document is larger than the preset storage space threshold value, the document needs to be split, so that the storage space of the plurality of split sub-documents is smaller than the preset storage space threshold value, and then the split sub-documents are stored. Specifically, when the document is split again, a CDC (Content-Defined Chunking) algorithm in the prior art may be adopted, or an FSP (Fixed-Sized Partition) algorithm may also be adopted, and the document is split according to the Content of the document, for example, the size of the storage space of the document to be stored is 1GB, the preset storage space threshold is 0.5GB, the document needs to be split before the document is stored, the document is split into two documents of 0.5GB by adopting the FSP algorithm, and then the two split sub-documents are stored in the corresponding storage servers according to the determined storage path. And when the user needs to read the complete document, downloading all the sub-documents according to the storage path, and merging the sub-documents according to the content of the document.
S103, storing the document into a first storage server as a main document according to a first storage path.
Specifically, a first storage server and a second storage server for storing documents are determined according to the performance index and the storage space, the documents to be stored are stored into the first storage server in a folder format according to a first storage path, and the documents in the first storage server are used as main documents, wherein if the format of the documents is in the form of a folder or a single file, the documents are directly stored in the first storage server; if the format of the document is in the form of a compressed package, the document in the form of the compressed package is decompressed and stored before being stored in the first storage server, the form of the main document is always kept and stored in the first storage server in the form of a folder or an individual document, and the first storage path is stored, so that a user can read the document without decompressing the document when using the document.
And S104, compressing the document and storing the document into a second storage server according to a second storage path to be used as a secondary document.
Specifically, in the process of storing the document in this embodiment, while the document is stored in the first storage server according to the first storage path, the document is compressed by using the compression software in the prior art, such as 360 compression software, and the compressed document is stored in the second storage server according to the determined second storage path as the secondary document, that is, the secondary document is stored in the form of a compressed package, and then the document is stored according to the second storage path, and the second storage path is stored, so that the user can quickly find the storage location of the secondary document.
S105, if the first storage server fails, decompressing the auxiliary document to obtain a main document; and determining a third storage server and a third storage path corresponding to the third storage server according to the storage space occupied by the main document and a preset storage threshold, compressing the main document and storing the main document into the third storage server according to the third storage path, wherein the preset storage threshold is determined by the total number of the storage servers in the current local area network and the performance index of each storage server.
Specifically, if the sentinel server monitors that the first storage server fails, for example, the first storage server fails to operate normally due to damage of a hard disk of the first storage server or existence of dangerous bugs, viruses and the like in a system, and the main document in the first storage server cannot be read, the sentinel server immediately sends an alarm signal to the processor, the processor decompresses the auxiliary document in the second storage server to make the auxiliary document become the main document, then identifies the storage space occupied by the main document, selects a third storage server and a third storage path which are in accordance with the storage space and the preset storage threshold in the current local area network, compresses the main document, and stores the compressed file in the third storage server as the auxiliary document according to the third storage path. By monitoring the working state of the storage server in real time, once a system fault occurs, other storage servers can be immediately searched, so that the copy of the document is saved, the security of document storage is improved, and the aim of automatically deploying the copy of the document is fulfilled.
It should be noted that, in step S103, the document is stored in the first storage server according to the first storage path as the main document, and in step S104, the document is compressed and stored in the second storage server according to the second storage path as the sub document. In an actual application process, the documents are stored in the first storage server and the second storage server without a sequential relationship, and the two storage servers are performed simultaneously, in this embodiment, fig. 1 is only used to illustrate that the documents are stored in the first storage server first and then in the second storage server, and are stored in the second storage server first and then in the first storage server, which is not shown in fig. 1.
According to the technical scheme, firstly, the storage space occupied by the document is identified according to the document uploaded by a user; selecting a storage server which accords with the storage document according to the performance indexes of the storage server nodes and the storage space and determining the storage path of the document in the storage server, and selecting the optimal storage path for the document so as to improve the storage efficiency of the document; storing the documents according to a storage path, so that a user can conveniently and quickly find the storage position of the documents, wherein the main document is stored in a first storage server, and the auxiliary document is stored in a second storage server; and if the first storage server fails, decompressing the secondary document to become the primary document, reselecting a proper third storage server again, and compressing the primary document to serve as a new secondary document so as to ensure the safety and reliability of document storage. Through the technical scheme, in the document storage process, a proper storage path can be selected for document storage, the storage efficiency is improved, and a safe and reliable storage service is provided for a user.
Fig. 2 is a flowchart of a distributed storage method according to another embodiment of the present invention, which is a refinement based on the foregoing embodiment. Referring to fig. 2, the method may specifically include the following steps:
s201, receiving a document storage request of a user, and acquiring the name of a document through the document storage request to determine the document to be stored.
In the storage process of the actual application, the distributed storage method provided in this embodiment is described by taking a storage server as an example, and the storage device is not limited herein. Specifically, when a user needs to store document data on a storage server, firstly, when the user stores a document, the user mainly clicks a window for uploading the document through a mouse, or a mobile phone is connected with the storage server through a data line or a bluetooth, the user sends the document to be stored to the storage server by clicking a sending button on a screen of the mobile phone, a processor of the storage server receives a request of the user for storing the document according to the above operations, and then identifies the input text information by using an OCR (Optical Character Recognition) technology according to the request for storing the document to determine the name, format and other information of the document to be stored.
S202, identifying the storage space occupied by the document to be stored.
S203, acquiring performance indexes of storage nodes of each storage server, matching the storage space with the performance indexes, and determining a first storage server and a second storage server, as well as a first storage path corresponding to the first storage server and a second storage path corresponding to the second storage server, wherein the number of the first storage server and the second storage server is at least one.
And S204, storing the document into a first storage server as a main document according to the first storage path.
S205, encrypting the main document according to an encryption algorithm.
Specifically, after the document to be stored is stored in the first storage server, the stored main document may be encrypted by using an encryption algorithm, the encryption algorithm may specifically adopt a symmetric key encryption algorithm in the prior art, that is, the same key is used in encryption and decryption, when the user reads the main document, the user needs to input the key to access the main document, and by encrypting the document, the security and confidentiality of the document may be improved, and a safe and reliable storage service may be provided for the user.
S206, compressing the document and storing the document into a second storage server as a secondary document according to a second storage path.
And S207, monitoring the working state of the first storage server in real time.
Specifically, in the actual application process, the working state of the storage server may have a plurality of conditions, for example, a normal working state and an abnormal working state, when the working state of the storage server is the normal working state, the security of document storage may not be threatened, and once the working state of the storage server is abnormal, the document in the failed storage server may face a risk of being lost, especially, some important document materials may be irreparably lost. Therefore, the working state of the storage server needs to be monitored in real time, and in this embodiment, the working state of the storage server is monitored mainly through the sentinel server, so that when the storage server fails, protective measures can be taken in time to prevent the document from being lost.
S208, if the first storage server fails, decompressing the auxiliary document to obtain a main document; and determining a third storage server and a third storage path corresponding to the third storage server according to the storage space occupied by the main document and a preset storage threshold, compressing the main document and storing the main document to the third storage server according to the third storage path, wherein the preset storage threshold is determined by the total number of the storage servers in the current local area network and the performance indexes of the storage servers.
It should be noted that in the above step S204, the document is stored into the first storage server according to the first storage path as the main document, and in the step S206, the document is compressed and stored into the second storage server according to the second storage path as the sub document. In an actual application process, the documents are stored in the first storage server and the second storage server without a sequential relationship, and the two storage servers are performed simultaneously, in this embodiment, fig. 2 is only used to illustrate that the documents are stored in the first storage server first and then in the second storage server, and are stored in the second storage server first and then in the first storage server, which is not shown in fig. 2.
Through the technical scheme, the specific information of each document to be stored can be accurately identified in the document storage process, and after the document is stored, the document is encrypted by adopting an encryption algorithm, so that the document storage safety can be improved, and a safe and reliable storage service can be provided for a user. In addition, the working state of the storage server is monitored in real time, so that the safety of document storage is further improved, and the document is prevented from being lost.
Fig. 3 is a flowchart of a distributed storage method according to another embodiment of the present invention, which is a refinement based on the foregoing embodiment. Referring to fig. 3, the method may specifically include the following steps:
s301, identifying a storage space occupied by the document to be stored.
S302, acquiring performance indexes of storage nodes of each storage server, matching storage space with the performance indexes according to a storage weight algorithm and a preset index weight threshold, and determining a first storage server and a second storage server, as well as a first storage path corresponding to the first storage server and a second storage path corresponding to the second storage server, wherein the number of the first storage server and the second storage server is at least one.
Specifically, in the process of storing a document, an optimal storage server and storage path may be selected for the document to be stored according to the size of the storage space occupied by the document to be stored, in this embodiment, after the storage space occupied by the document to be stored and the performance indexes of the storage nodes of each storage server are obtained, the storage space occupied by the document to be stored and the performance indexes of the storage nodes of each storage server may be compared by a storage cloud center in a processor according to a weight algorithm and a preset index weight threshold in the prior art, for example, the storage space occupied by the document to be stored and the performance indexes of the storage nodes of each storage server are 1GB, the storage cloud center takes two storage servers as an example, and the storage servers are 0.9GB and 1.5GB, and finally selects a storage server of 1.5GB according to the weight algorithm and the index weight threshold.
In this embodiment, according to a storage weight algorithm and a preset index weight threshold, a first storage server and a second storage server need to be determined, and in an actual application process, the number of the first storage server and the second storage server is at least one, but in this embodiment, only one first storage server and one second storage server are used for description, and no limitation is made to this, where the first storage server is used for storing a main document, and the second storage server is used for storing a secondary document. By adopting a storage weight algorithm and a preset index weight threshold value, the storage space is matched with the performance index, the most appropriate storage path can be automatically selected for the storage of the document in the document storage process, the storage efficiency is improved, and the document storage safety can be improved by storing the document in different storage servers.
S303, storing the document into a first storage server according to a first storage path to be used as a main document.
S304, compressing the document and storing the document into a second storage server according to a second storage path to be used as a secondary document.
S305, if the first storage server fails, decompressing the auxiliary document to obtain a main document; if the storage space occupied by the main document is smaller than a preset storage amount threshold value, determining a third storage server in the current local area network; and if the storage space occupied by the main document is larger than the preset storage volume threshold, determining a third storage server outside the current local area network and a third storage path corresponding to the third storage server, compressing the main document and storing the main document to the third storage server according to the third storage path, wherein the preset storage volume threshold is determined by the total number of the storage servers in the current local area network and the performance indexes of all the storage servers.
Specifically, in this embodiment, the sentinel server monitors the working state of the first storage server storing the main document in real time, and when it is monitored that the first storage server fails, the sentinel server decompresses the sub document in the second storage server in time, and uses the decompressed document as a new main document. At this time, since the secondary document is decompressed into the primary document and no other secondary document exists, in order to ensure the security of document storage, the third storage server needs to be determined again according to the size of the storage space occupied by the primary document for storing the secondary document, and for selecting the third storage server, the third storage server needs to be determined according to the preset storage amount threshold value in the current local area network, wherein the preset storage amount threshold value is determined by the total number of storage servers in the current local area network and the performance index of each storage server. For example, if there is still another storage server in the current local area network and the remaining storage amount of the remaining storage server is 1GB, the preset storage amount threshold is 1GB, the storage space occupied by the main document is 0.5GB, and at this time, the storage space is smaller than the preset storage amount threshold, the third storage server is preferentially determined in the current local area network; if no other storage server can store the document in the current local area network, or the residual storage amount is 0.3GB, namely the threshold value of the preset storage amount is 0.3GB, and the storage space at the moment is larger than the threshold value of the preset storage amount, a third storage server needs to be determined outside the current local area network, and the document storage safety is ensured by automatically deploying the document copy in different places.
It should be noted that in S303, the document is stored into the first storage server according to the first storage path as a main document, and in S304, the document is compressed and stored into the second storage server according to the second storage path as a sub document. In an actual application process, the documents are stored in the first storage server and the second storage server without a sequential relationship, and the two storage servers are performed simultaneously, in this embodiment, fig. 3 is only used to illustrate that the documents are stored in the first storage server first and then in the second storage server, and are stored in the second storage server first and then in the first storage server, which is not shown in fig. 3.
According to the technical scheme provided by the embodiment, in the document storage process, the storage space is matched with the performance index according to the storage weight algorithm and the preset index weight threshold value, the first storage server and the second storage server are determined, the optimal storage path can be selected for document storage, and the document storage efficiency can be improved; if the first storage server fails, whether a third storage server is determined in the current local area network or not can be determined according to a preset storage threshold value in the current local area network, the purpose that local or allopatric copies can be automatically deployed in document storage is achieved, dynamic expansion storage is supported in the document storage process, and a storage space capable of being automatically expanded and safe and reliable storage service is provided for a user.
Further, the performance indicators include: hard disk capacity, the number of central processing units, the processing speed of the central processing units and the success rate of historical storage.
Specifically, in this embodiment, the performance index of the storage node of each storage server obtained by the registration monitoring center may be the hard disk capacity of the storage server, and the larger the hard disk capacity is, the more documents are stored; the performance index can be the number of central processing units in the storage server, and in the actual application process, the greater the number of the central processing units, the faster the processing speed and the higher the efficiency; the performance index can also be the processing speed of a central processing unit in the storage server, and the higher the speed of processing data is, the higher the storage efficiency is; the performance index may also be a history storage success rate, and the storage of the document on the storage server may fail due to different performances of the storage server, so that the history storage success rate of the storage server is considered to be part of improving the storage efficiency.
Fig. 4 is a schematic structural diagram of a distributed storage apparatus according to another embodiment of the present invention, which is suitable for executing a distributed storage method according to an embodiment of the present invention. As shown in fig. 4, the apparatus may specifically include: an identification module 401, a determination module 402, a first storage module 403, a second storage module 404, and a control module 405, wherein;
the identifying module 401 is configured to identify a storage space occupied by a document to be stored.
A determining module 402, configured to obtain performance indexes of storage nodes of each storage server, match the storage space with the performance indexes, and determine a first storage server and a second storage server, as well as a first storage path corresponding to the first storage server and a second storage path corresponding to the second storage server, where the number of the first storage server and the second storage server is at least one.
The first storage module 403 is configured to store the document into the first storage server according to the first storage path as a main document.
And a second storage module 404, configured to compress the document and store the document in a second storage server as a secondary document according to a second storage path.
The control module 405 is configured to decompress the secondary document to obtain the primary document if the first storage server fails; and determining a third storage server and a third storage path corresponding to the third storage server according to the storage space occupied by the main document and a preset storage threshold, compressing the main document and storing the main document to the third storage server according to the third storage path, wherein the preset storage threshold is determined by the total number of the storage servers in the current local area network and the performance indexes of the storage servers.
The document storage system further comprises an information acquisition module, which is used for receiving a document storage request of a user before identifying the storage space occupied by the document to be stored, and acquiring the name of the document through the document storage request to determine the document to be stored.
Further, the determining module 402 includes: and the matching module is specifically used for matching the storage space with the performance index according to the storage weight algorithm and a preset index weight threshold.
The document encryption method further comprises an encryption module, which is used for encrypting the main document according to an encryption algorithm after the document is stored into the first storage server according to the first storage path to serve as the main document.
The monitoring module is used for monitoring the working state of the first storage server in real time before the first storage server fails.
Further, the control module 405 includes: the judging module is specifically used for determining a third storage server in the current local area network if the storage space occupied by the main document is smaller than a preset storage threshold; and if the storage space occupied by the main document is larger than the preset storage volume threshold, determining a third storage server outside the current local area network.
Further, the performance indicators include: hard disk capacity, the number of central processing units, the processing speed of the central processing units and the success rate of historical storage.
The distributed storage device provided by the embodiment of the invention can execute the distributed storage method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
Fig. 5 is a schematic structural diagram of a terminal device according to another embodiment of the present invention, and as shown in fig. 5, the terminal device may include:
a processor 501, and a memory 502 connected to the processor 501; the memory 502 is used to store a computer program for executing the distributed storage method described above; the processor 501 is used for calling and executing the computer program in the memory 502; the distributed storage method at least comprises the following steps:
identifying a storage space occupied by a document to be stored; the method comprises the steps of obtaining performance indexes of storage nodes of each storage server, matching storage space with the performance indexes, and determining a first storage server and a second storage server, as well as a first storage path corresponding to the first storage server and a second storage path corresponding to the second storage server, wherein the number of the first storage server and the second storage server is at least one; storing the document into a first storage server according to a first storage path to be used as a main document; compressing the document and storing the document into a second storage server according to a second storage path to be used as a side document; if the first storage server fails, decompressing the secondary document to obtain a primary document; and determining a third storage server and a third storage path corresponding to the third storage server according to the storage space occupied by the main document and a preset storage threshold, compressing the main document and storing the main document to the third storage server according to the third storage path, wherein the preset storage threshold is determined by the total number of the storage servers in the current local area network and the performance indexes of the storage servers.
The embodiment of the present invention may further include a storage medium, where the storage medium stores a computer program, and when the computer program is executed by the processor 501, the method implements the steps in the distributed storage method, where the steps in the method specifically include:
identifying a storage space occupied by a document to be stored; the method comprises the steps of obtaining performance indexes of storage nodes of each storage server, matching storage space with the performance indexes, and determining a first storage server and a second storage server, as well as a first storage path corresponding to the first storage server and a second storage path corresponding to the second storage server, wherein the number of the first storage server and the second storage server is at least one; storing the document into a first storage server according to a first storage path to be used as a main document; compressing the document and storing the document into a second storage server according to a second storage path to be used as a side document; if the first storage server fails, decompressing the auxiliary document to obtain a main document; and determining a third storage server and a third storage path corresponding to the third storage server according to the storage space occupied by the main document and a preset storage threshold, compressing the main document and storing the main document to the third storage server according to the third storage path, wherein the preset storage threshold is determined by the total number of the storage servers in the current local area network and the performance indexes of the storage servers.
It is understood that the same or similar parts in the above embodiments may be mutually referred to, and the same or similar parts in other embodiments may be referred to for the content which is not described in detail in some embodiments.
It should be noted that the terms "first," "second," and the like in the description of the present invention are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. Further, in the description of the present invention, the meaning of "a plurality" means at least two unless otherwise specified.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and alternate implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following technologies, which are well known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.
In addition, functional units in the embodiments of the present invention may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.
The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc., and the SSD solid state disk is recommended.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.
Claims (10)
1. A distributed storage method, comprising:
identifying a storage space occupied by a document to be stored;
acquiring performance indexes of storage nodes of each storage server, matching the storage space with the performance indexes, and determining a first storage server and a second storage server, as well as a first storage path corresponding to the first storage server and a second storage path corresponding to the second storage server, wherein the number of the first storage server and the second storage server is at least one;
storing the document into the first storage server according to the first storage path to serve as a main document;
compressing the document and storing the document into the second storage server according to the second storage path to be used as a secondary document;
if the first storage server fails, decompressing the secondary document to obtain the primary document; and determining a third storage server and a third storage path corresponding to the third storage server according to the storage space occupied by the main document and a preset storage threshold, compressing the main document and storing the main document to the third storage server according to the third storage path, wherein the preset storage threshold is determined by the total number of the storage servers in the current local area network and the performance indexes of the storage servers.
2. The distributed storage method according to claim 1, before identifying the storage space occupied by the document to be stored, further comprising:
receiving a document storage request of a user, and acquiring the name of the document through the document storage request to determine the document to be stored.
3. The distributed storage method of claim 1, wherein said matching the storage space to the performance indicator comprises:
and matching the storage space with the performance index according to a storage weight algorithm and a preset index weight threshold.
4. The distributed storage method according to claim 1, after storing the document into the first storage server as a primary document according to the first storage path, further comprising:
and encrypting the main document according to an encryption algorithm.
5. The distributed storage method according to claim 1, further comprising, before the failure of the first storage server:
and monitoring the working state of the first storage server in real time.
6. The distributed storage method according to claim 1, wherein the determining a third storage server according to the storage space occupied by the main document and a preset storage threshold comprises:
if the storage space occupied by the main document is smaller than the preset storage space threshold, determining a third storage server in the current local area network;
and if the storage space occupied by the main document is larger than the preset storage threshold, determining the third storage server outside the current local area network.
7. The distributed storage method of claim 1, wherein the performance indicators comprise: hard disk capacity, the number of central processing units, the processing speed of the central processing units and the success rate of historical storage.
8. A distributed storage apparatus, characterized in that, when applied to the distributed storage method according to any one of claims 1 to 7, the apparatus comprises:
the identification module is used for identifying the storage space occupied by the document to be stored;
the determining module is configured to acquire performance indexes of storage nodes of each storage server, match the storage space with the performance indexes, and determine a first storage server and a second storage server, as well as a first storage path corresponding to the first storage server and a second storage path corresponding to the second storage server, where the number of the first storage server and the second storage server is at least one;
the first storage module is used for storing the document into the first storage server according to the first storage path to be used as a main document;
the second storage module is used for compressing the document and storing the document into the second storage server according to the second storage path to serve as a secondary document;
the control module is used for decompressing the auxiliary document to obtain the main document if the first storage server fails; and determining a third storage server and a third storage path corresponding to the third storage server according to the storage space occupied by the main document and a preset storage threshold, compressing the main document and storing the main document to the third storage server according to the third storage path, wherein the preset storage threshold is determined by the total number of the storage servers in the current local area network and the performance indexes of the storage servers.
9. A terminal device, comprising:
a processor, and a memory coupled to the processor;
the memory for storing a computer program for performing at least the distributed storage method of any one of claims 1-7;
the processor is used for calling and executing the computer program in the memory.
10. A storage medium, characterized in that the storage medium stores a computer program which, when executed by a processor, implements the steps in the distributed storage method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910730894.2A CN110413237B (en) | 2019-08-08 | 2019-08-08 | Distributed storage method, device, terminal equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910730894.2A CN110413237B (en) | 2019-08-08 | 2019-08-08 | Distributed storage method, device, terminal equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110413237A CN110413237A (en) | 2019-11-05 |
| CN110413237B true CN110413237B (en) | 2022-12-06 |
Family
ID=68366746
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910730894.2A Active CN110413237B (en) | 2019-08-08 | 2019-08-08 | Distributed storage method, device, terminal equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110413237B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114281260A (en) * | 2021-12-23 | 2022-04-05 | 广州炒米信息科技有限公司 | Storage method, device, equipment and medium applied to distributed storage system |
| CN117130566A (en) * | 2023-10-27 | 2023-11-28 | 睿至科技集团有限公司 | Distributed storage method and storage platform |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5696926A (en) * | 1993-07-30 | 1997-12-09 | Apple Computer, Inc. | Method and apparatus for transparently compressing data in a primary storage device |
| CN109582245A (en) * | 2018-12-06 | 2019-04-05 | 联想(北京)有限公司 | Data processing method, device and equipment |
-
2019
- 2019-08-08 CN CN201910730894.2A patent/CN110413237B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5696926A (en) * | 1993-07-30 | 1997-12-09 | Apple Computer, Inc. | Method and apparatus for transparently compressing data in a primary storage device |
| CN109582245A (en) * | 2018-12-06 | 2019-04-05 | 联想(北京)有限公司 | Data processing method, device and equipment |
Non-Patent Citations (1)
| Title |
|---|
| 企业内部分布式存储高效扩展机制研究;张金波;《计算机工程与科学》;20150430(第04期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110413237A (en) | 2019-11-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109597717B (en) | Data backup and recovery method and device, electronic equipment and storage medium | |
| US9244790B1 (en) | System and method for predicting future disk failures | |
| US9582194B2 (en) | Techniques for improving performance of a backup system | |
| US9454321B1 (en) | Workload-driven storage configuration management | |
| CN108509153B (en) | OSD selection method, data writing and reading method, monitor and server cluster | |
| US8452731B2 (en) | Remote backup and restore | |
| CN103095843B (en) | A kind of data back up method and client based on version vector | |
| US8874520B2 (en) | Processes and methods for client-side fingerprint caching to improve deduplication system backup performance | |
| JP5774794B2 (en) | Storage system and storage system control method | |
| US8825626B1 (en) | Method and system for detecting unwanted content of files | |
| US20200073596A1 (en) | Data structure storage and data management | |
| US8806062B1 (en) | Adaptive compression using a sampling based heuristic | |
| CN110413237B (en) | Distributed storage method, device, terminal equipment and storage medium | |
| WO2013027230A1 (en) | Storage apparatus and duplicate data detection method | |
| US10091293B2 (en) | Rapid cloud-based image centralization | |
| CN111262822B (en) | File storage method, device, block link point and system | |
| CN110651258B (en) | Automatic backup device based on real-time file change detection | |
| US9451024B2 (en) | Self-organizing disk (SoD) | |
| US11934280B2 (en) | Use of cluster-level redundancy within a cluster of a distributed storage management system to address node-level errors | |
| CN105095027A (en) | Data backup method and apparatus | |
| CN110389715B (en) | Data storage method, storage server and cloud storage system | |
| JP6943008B2 (en) | Control programs, control methods, and information processing equipment | |
| CN105159790A (en) | Data rescue method and file server | |
| US20180246666A1 (en) | Methods for performing data deduplication on data blocks at granularity level and devices thereof | |
| US11157381B2 (en) | Display control method and display control device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |