CN112416265A - Storage node space capacity and security evaluation method and system based on big data - Google Patents
Storage node space capacity and security evaluation method and system based on big data Download PDFInfo
- Publication number
- CN112416265A CN112416265A CN202011491797.1A CN202011491797A CN112416265A CN 112416265 A CN112416265 A CN 112416265A CN 202011491797 A CN202011491797 A CN 202011491797A CN 112416265 A CN112416265 A CN 112416265A
- Authority
- CN
- China
- Prior art keywords
- storage node
- data
- module
- storage
- safety
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
- G06F3/0623—Securing storage systems in relation to content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0662—Virtualisation aspects
- G06F3/0667—Virtualisation aspects at data level, e.g. file, record or object virtualisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention belongs to the technical field of data security, and discloses a storage node space capacity and security assessment method and system based on big data, which comprises the following steps: acquiring basic information and state data of storage nodes, and screening currently available data storage nodes; calculating the safety factor of each available data storage node, and evaluating the performance of the storage node; predicting and calculating the failure rate of each storage node; carrying out safety performance evaluation on each storage node; sequencing according to the safety performance evaluation result, and generating a storage reference table for feedback according to the sequencing result of each storage node and the size of the available storage space of the corresponding storage node; and utilizing the storage nodes through the calling module according to the storage reference table. The invention effectively reduces the operation amount of data and improves the evaluation efficiency; the accuracy and comprehensiveness of evaluation are improved, and the safety and reliability of data storage are further improved.
Description
Technical Field
The invention belongs to the technical field of data security, and particularly relates to a storage node space capacity and security assessment method and system based on big data.
Background
Currently, Cloud storage is a mode of online storage (english: Cloud storage), i.e., data is stored on multiple virtual servers, usually hosted by third parties, rather than on dedicated servers. Hosting companies operate large data centers, and people who need data storage hosting meet the data storage requirements by buying or leasing storage space. The data center operator prepares the storage virtualized resources at the back end according to the needs of the user, and provides the resources in a storage resource pool (storage pool), so that the user can use the storage resource pool to store the files or objects. In practice, these resources may be distributed over numerous server hosts.
The prior art mainly focuses on load balancing aiming at cloud storage, namely how to reasonably distribute loads to each node; or further, the data is stored in a distributed mode by encrypting the data, namely, the reliability of the data is improved, so that the safety is enhanced.
However, in the prior art, a method for judging or evaluating the security of the storage node by paying attention to the performance, the security factor and the like of the storage node is not provided.
Through the above analysis, the problems and defects of the prior art are as follows: in the prior art, the safety evaluation result of the storage node only depends on load factors, internal factors are not considered, and the evaluation is inaccurate.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a storage node space capacity and security assessment method and system based on big data.
The invention is realized in such a way that a method for evaluating the space capacity and the safety of a storage node based on big data comprises the following steps:
acquiring the type, the number, the original data volume and other relevant basic information of storage nodes through a basic information acquisition module; acquiring relevant state data of the storage node through a state data acquisition module; screening currently available data storage nodes based on the acquired relevant state information of the storage nodes through a screening module;
secondly, controlling a safety coefficient calculation module by using a single chip microcomputer or a controller through a central control module to calculate the safety coefficient of each available data storage node; evaluating the performance of the storage node based on the acquired relevant state data of the storage node through a performance evaluation module;
thirdly, predicting and calculating the fault rate of each storage node through a fault rate prediction module; the safety performance of each storage node is evaluated through a safety evaluation module based on basic information, state data, safety factors, performance evaluation results and failure rate of the storage nodes;
sequencing according to the safety performance evaluation result through a feedback module, and generating a storage reference table for the sequencing result of each storage node and the size of the available storage space of the corresponding storage node to feed back; utilizing the storage nodes through a calling module according to the storage reference table;
in the second step, the calculating the safety factor of the storage node by the safety factor calculating module includes:
(1) acquiring defect parameters and risk parameters of storage nodes, and establishing a safety coefficient generation model based on a radial basis function neural network based on the acquired defect parameters and risk parameters of the storage nodes;
(2) determining the calculation weight of the defect parameters and the risk parameters in the safety coefficient generation model according to the historical defect parameters and the historical risk parameters;
(3) generating the safety coefficient of the storage node by utilizing the corresponding calculation weight of the defect parameter and the risk parameter and a pre-established safety coefficient generation model based on the radial basis function neural network;
in the first step, the specific process of the screening module screening the currently available data storage nodes based on the acquired relevant state information of the storage nodes is as follows:
establishing a corresponding data set according to the relevant state information of the storage nodes;
scanning data in the data set, calculating the distance between each data object and an adjacent object, accumulating to calculate the distance sum, and calculating the distance sum average value;
assuming that the sum of the distances of a certain data object is greater than the sum of the distances and the mean value, regarding the point as a storage node; moving the object from the data set to the storage node set, and repeating until all storage nodes are found;
in the second step, the evaluating the performance of the storage node by the performance evaluating module based on the acquired relevant state data of the storage node includes:
firstly, acquiring relevant state data of a storage node; calculating the residual performance value and the maximum occupied bandwidth value according to the load rate and the original performance value in the state data of each storage node;
then, obtaining the total residual performance value of the storage system according to the residual performance value of each unfilled storage node;
and finally, judging the performance of the storage nodes based on the performance allowance rate and the maximum occupied bandwidth value according to the ratio of the residual performance value of each storage node to the total residual performance value of all the storage nodes as the performance allowance rate.
Further, in the step (1), the establishing of the safety coefficient generation model based on the radial basis function neural network based on the acquired defect parameters and risk parameters of the storage nodes includes:
training the safety coefficient generation model based on the radial basis function neural network by using the historical defect parameters and the historical risk parameters; and correcting the trained safety coefficient generation model based on the radial basis function neural network by using the historical defect parameters and the historical risk parameters.
Further, the method for calculating the maximum occupied bandwidth value comprises the following steps:
determining a calculation mode of occupied bandwidth of a storage node in a data read-write scene; based on the determined calculation mode of the occupied bandwidth of each node, estimating the occupied bandwidth of each node in a data reading and writing scene according to the original data quantity of the storage nodes; and determining the maximum occupied bandwidth in the occupied bandwidths of the nodes.
Further, the judging the performance of the storage node based on the performance allowance rate and the maximum occupied bandwidth value includes:
the larger the performance allowance rate is, the better the performance state is; a smaller maximum occupied bandwidth value represents a better performance state.
Further, the determining the performance of the storage node based on the performance allowance rate and the maximum occupied bandwidth value further includes:
and carrying out homodromous processing on the performance allowance rate and the maximum occupied bandwidth value, respectively setting weight values of the performance allowance rate and the maximum occupied bandwidth value in the performance evaluation of the storage node, and evaluating the performance of the storage node based on the weight values and the obtained homodromous processing performance allowance rate and maximum occupied bandwidth value data.
Further, in the third step, the pre-estimating and calculating the failure rate of each storage node by the failure rate pre-estimating module includes:
determining an upper limit value of the read-write erasing use times of each storage node based on different storage node media types, and determining the fault probability of the storage nodes based on the obtained input/output read-write information, the input/output abnormal information and the upper limit value of different blocks of each storage node.
Further, in step three, the performing, by the security evaluation module, security performance evaluation of each storage node based on the basic information, the state data, the security coefficient, the performance evaluation result, and the failure rate of the storage node includes:
1) basic information, state data, safety coefficients, performance evaluation results and fault rates of the storage nodes are used as safety evaluation indexes of the storage nodes, and corresponding weights are distributed to the safety evaluation indexes;
2) calculating the score of the safety evaluation index, and carrying out weighted average on the score of the safety evaluation index based on the weight of the safety evaluation index to obtain the score of the safety evaluation index;
3) and comparing the safety score with a preset safety evaluation standard to obtain an evaluation target safety evaluation conclusion.
Another object of the present invention is to provide a big data based storage node space capacity and security evaluation system implementing the big data based storage node space capacity and security evaluation method, the big data based storage node space capacity and security evaluation system comprising:
the system comprises a basic information acquisition module, a state data acquisition module, a screening module, a central control module, a safety coefficient calculation module, a performance evaluation module, a fault rate estimation module, a safety evaluation module, a feedback module and a calling module;
the basic information acquisition module is connected with the central control module and is used for acquiring the type, the number, the original data volume and other related basic information of the storage nodes;
the state data acquisition module is connected with the central control module and used for acquiring the related state data of the storage nodes;
the screening module is connected with the central control module and used for screening available data storage nodes based on the acquired relevant state information of the storage nodes; the specific process of screening the currently available data storage node by the screening module based on the acquired relevant state information of the storage node is as follows: establishing a corresponding data set according to the relevant state information of the storage nodes; scanning data in the data set, calculating the distance between each data object and an adjacent object, accumulating to calculate the distance sum, and calculating the distance sum average value; assuming that the sum of the distances of a certain data object is greater than the sum of the distances and the mean value, regarding the point as a storage node; moving the object from the data set to the storage node set, and repeating until all storage nodes are found;
and the central control module is connected with the basic information acquisition module, the state data acquisition module, the screening module, the safety coefficient calculation module, the performance evaluation module, the fault rate estimation module, the safety evaluation module, the feedback module and the calling module and is used for controlling each module to normally work by utilizing a single chip microcomputer or a controller.
Further, the state information of the data storage node includes:
the load rate, the original performance value, the input/output read-write information, the input/output type abnormal information and other related information of the storage node are obtained.
Further, the big data based storage node space capacity and security evaluation system further comprises:
the safety coefficient calculation module is connected with the central control module and used for calculating the safety coefficient of each available data storage node; acquiring defect parameters and risk parameters of storage nodes, and establishing a safety coefficient generation model based on a radial basis function neural network based on the acquired defect parameters and risk parameters of the storage nodes; determining the calculation weight of the defect parameters and the risk parameters in the safety coefficient generation model according to the historical defect parameters and the historical risk parameters; generating the safety coefficient of the storage node by utilizing the corresponding calculation weight of the defect parameter and the risk parameter and a pre-established safety coefficient generation model based on the radial basis function neural network;
the performance evaluation module is connected with the central control module and used for evaluating the performance of the storage nodes based on the acquired relevant state data of the storage nodes;
the failure rate prediction module is connected with the central control module and used for predicting and calculating the failure rate of each storage node; determining an upper limit value of the read-write erasing use times of each storage node based on different storage node media types, and determining the fault probability of the storage nodes based on the obtained input and output read-write information, input and output abnormal information and the upper limit value of different blocks of each storage node;
the safety evaluation module is connected with the central control module and is used for evaluating the safety performance of each storage node based on the basic information, the state data, the safety coefficient, the performance evaluation result and the fault rate of the storage node; the safety performance evaluation module carries out safety performance evaluation on each storage node based on the basic information, the state data, the safety factor, the performance evaluation result and the failure rate of the storage node, and comprises the following steps: basic information, state data, safety coefficients, performance evaluation results and fault rates of the storage nodes are used as safety evaluation indexes of the storage nodes, and corresponding weights are distributed to the safety evaluation indexes; calculating the score of the safety evaluation index, and carrying out weighted average on the score of the safety evaluation index based on the weight of the safety evaluation index to obtain the score of the safety evaluation index; comparing the safety score with a preset safety evaluation standard to obtain an evaluated target safety evaluation conclusion;
the feedback module is connected with the central control module and used for sequencing according to the safety performance evaluation result and generating a storage reference table for feeding back the sequencing result of each storage node and the size of the available storage space of the corresponding storage node;
and the calling module is connected with the central control module and is used for utilizing the storage nodes according to the storage reference table.
By combining all the technical schemes, the invention has the advantages and positive effects that: according to the invention, the available storage nodes are screened by acquiring the basic information of the storage nodes, so that the data calculation amount is effectively reduced, and the evaluation efficiency is improved; the safety and the space capacity of the storage nodes are integrally evaluated through essential parameters of the storage nodes such as basic information, performance, safety factors, failure rate and the like, the accuracy and the comprehensiveness of evaluation are improved, and the safety and the reliability of data storage are further improved.
The invention is used for collecting the type, the quantity, the original data volume and other relevant basic information of the storage node through the basic information collection module; the state data acquisition module is used for acquiring relevant state data of the storage nodes;
the screening module is used for screening available data storage nodes based on the acquired relevant state information of the storage nodes; the safety coefficient calculation module is used for calculating the safety coefficient of each available data storage node; the performance evaluation module is used for evaluating the performance of the storage node based on the acquired relevant state data of the storage node; the failure rate estimation module is used for estimating and calculating the failure rate of each storage node; the safety evaluation module is used for evaluating the safety performance of each storage node based on the basic information, the state data, the safety coefficient, the performance evaluation result and the failure rate of the storage node; the feedback module is used for sequencing according to the safety performance evaluation result and generating a storage reference table for feeding back the sequencing result of each storage node and the size of the available storage space of the corresponding storage node; and the calling module is used for utilizing the storage nodes according to the storage reference table.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained from the drawings without creative efforts.
Fig. 1 is a flowchart of a method for evaluating space capacity and security of a storage node based on big data according to an embodiment of the present invention.
Fig. 2 is a flowchart of a method for calculating a security factor of a storage node by a security factor calculation module according to an embodiment of the present invention.
Fig. 3 is a flowchart of a method for evaluating the performance of a storage node by a performance evaluation module according to an embodiment of the present invention, based on acquired relevant state data of the storage node.
Fig. 4 is a flowchart of a method for evaluating the security performance of each storage node by the security evaluation module based on the basic information, the state data, the security coefficient, the performance evaluation result, and the failure rate of the storage node according to the embodiment of the present invention.
FIG. 5 is a schematic structural diagram of a system for evaluating space capacity and security of a storage node based on big data according to an embodiment of the present invention;
in the figure: 1. a basic information acquisition module; 2. a status data acquisition module; 3. a screening module; 4. a central control module; 5. a safety coefficient calculation module; 6. a performance evaluation module; 7. a failure rate estimation module; 8. a security evaluation module; 9. a feedback module; 10. and calling the module.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In view of the problems in the prior art, the present invention provides a method and a system for evaluating the space capacity and security of a storage node based on big data, and the present invention is described in detail below with reference to the accompanying drawings.
As shown in fig. 1, a method for evaluating space capacity and security of a storage node based on big data according to an embodiment of the present invention includes:
s101, acquiring the type, the number, the original data volume and other related basic information of the storage nodes through a basic information acquisition module; acquiring relevant state data of the storage node through a state data acquisition module; screening currently available data storage nodes based on the acquired relevant state information of the storage nodes through a screening module;
s102, controlling a safety coefficient calculation module by a single chip microcomputer or a controller through a central control module to calculate the safety coefficient of each available data storage node; evaluating the performance of the storage node based on the acquired relevant state data of the storage node through a performance evaluation module;
s103, predicting and calculating the fault rate of each storage node through a fault rate prediction module; the safety performance of each storage node is evaluated through a safety evaluation module based on basic information, state data, safety factors, performance evaluation results and failure rate of the storage nodes;
s104, sorting according to the safety performance evaluation result through a feedback module, and generating a storage reference table for feeding back the sorting result of each storage node and the size of the available storage space of the corresponding storage node; and utilizing the storage nodes through the calling module according to the storage reference table.
In step S101, the specific process of the screening module provided in the embodiment of the present invention for screening the currently available data storage node based on the obtained relevant state information of the storage node is as follows:
establishing a corresponding data set according to the relevant state information of the storage nodes;
scanning data in the data set, calculating the distance between each data object and an adjacent object, accumulating to calculate the distance sum, and calculating the distance sum average value;
assuming that the sum of distances for a data object is greater than the sum of distances, the point is considered a storage node. This object is moved from the data set to the storage node set, iteratively until all storage nodes are found.
As shown in fig. 2, in step S102, the calculating the security factor of the storage node by the security factor calculating module according to the embodiment of the present invention includes:
s201, acquiring defect parameters and risk parameters of storage nodes, and establishing a safety coefficient generation model based on a radial basis function neural network based on the acquired defect parameters and risk parameters of the storage nodes;
s202, determining the calculation weight of the defect parameters and the risk parameters in the safety coefficient generation model according to the historical defect parameters and the historical risk parameters;
and S203, generating the safety coefficient of the storage node by using the corresponding calculation weight of the defect parameter and the risk parameter and a pre-established safety coefficient generation model based on the radial basis function neural network.
In step S201, the establishing of the safety coefficient generation model based on the radial basis function neural network based on the acquired defect parameters and risk parameters of the storage node according to the embodiment of the present invention includes:
training the safety coefficient generation model based on the radial basis function neural network by using the historical defect parameters and the historical risk parameters; and correcting the trained safety coefficient generation model based on the radial basis function neural network by using the historical defect parameters and the historical risk parameters.
As shown in fig. 3, in step S102, the evaluating the performance of the storage node by the performance evaluating module according to the embodiment of the present invention based on the acquired relevant state data of the storage node includes:
s301, acquiring relevant state data of the storage node; calculating the residual performance value and the maximum occupied bandwidth value according to the load rate and the original performance value in the state data of each storage node;
s302, obtaining a total residual performance value of the storage system according to the residual performance value of each incomplete storage node;
and S303, judging the performance of the storage nodes based on the performance tolerance rate and the maximum occupied bandwidth value according to the ratio of the residual performance value of each storage node to the total residual performance value of all the storage nodes as the performance tolerance rate.
The method for calculating the maximum occupied bandwidth value provided by the embodiment of the invention comprises the following steps:
determining a calculation mode of occupied bandwidth of a storage node in a data read-write scene; based on the determined calculation mode of the occupied bandwidth of each node, estimating the occupied bandwidth of each node in a data reading and writing scene according to the original data quantity of the storage nodes; and determining the maximum occupied bandwidth in the occupied bandwidths of the nodes.
The method for judging the performance of the storage node based on the performance allowance rate and the maximum occupied bandwidth value, provided by the embodiment of the invention, comprises the following steps:
the larger the performance allowance rate is, the better the performance state is; a smaller maximum occupied bandwidth value represents a better performance state.
The method for judging the performance of the storage node based on the performance allowance rate and the maximum occupied bandwidth value, provided by the embodiment of the invention, further comprises the following steps:
and carrying out homodromous processing on the performance allowance rate and the maximum occupied bandwidth value, respectively setting weight values of the performance allowance rate and the maximum occupied bandwidth value in the performance evaluation of the storage node, and evaluating the performance of the storage node based on the weight values and the obtained homodromous processing performance allowance rate and maximum occupied bandwidth value data.
In step S103, the predicting and calculating the failure rate of each storage node by the failure rate predicting module according to the embodiment of the present invention includes:
determining an upper limit value of the read-write erasing use times of each storage node based on different storage node media types, and determining the fault probability of the storage nodes based on the obtained input/output read-write information, the input/output abnormal information and the upper limit value of different blocks of each storage node.
As shown in fig. 4, in step S103, the performing, by the security evaluation module, security performance evaluation of each storage node based on the basic information, the state data, the security factor, the performance evaluation result, and the failure rate of the storage node according to the embodiment of the present invention includes:
s401, taking basic information, state data, safety factors, performance evaluation results and fault rates of the storage nodes as safety evaluation indexes of the storage nodes, and distributing corresponding weights to the safety evaluation indexes;
s402, calculating the score of the safety evaluation index, and carrying out weighted average on the score of the safety evaluation index based on the weight of the safety evaluation index to obtain the score of the safety evaluation index;
and S403, comparing the safety score with a preset safety evaluation standard to obtain an evaluation conclusion of the safety of the evaluated target.
As shown in fig. 5, a system for evaluating space capacity and security of a storage node based on big data according to an embodiment of the present invention includes:
the system comprises a basic information acquisition module 1, a state data acquisition module 2, a screening module 3, a central control module 4, a safety coefficient calculation module 5, a performance evaluation module 6, a fault rate estimation module 7, a safety evaluation module 8, a feedback module 9 and a calling module 10;
the basic information acquisition module 1 is connected with the central control module 4 and is used for acquiring the type, the number, the original data volume and other related basic information of the storage nodes;
the state data acquisition module 2 is connected with the central control module 4 and is used for acquiring the related state data of the storage nodes;
the screening module 3 is connected with the central control module 4 and used for screening available data storage nodes based on the acquired relevant state information of the storage nodes;
the central control module 4 is connected with the basic information acquisition module 1, the state data acquisition module 2, the screening module 3, the safety coefficient calculation module 5, the performance evaluation module 6, the fault rate estimation module 7, the safety evaluation module 8, the feedback module 9 and the calling module 10, and is used for controlling each module to normally work by utilizing a single chip microcomputer or a controller;
the safety coefficient calculation module 5 is connected with the central control module 4 and used for calculating the safety coefficient of each available data storage node;
the performance evaluation module 6 is connected with the central control module 4 and used for evaluating the performance of the storage nodes based on the acquired relevant state data of the storage nodes;
the failure rate estimation module 7 is connected with the central control module 4 and used for estimating and calculating the failure rate of each storage node;
the safety evaluation module 8 is connected with the central control module 4 and is used for evaluating the safety performance of each storage node based on the basic information, the state data, the safety coefficient, the performance evaluation result and the fault rate of the storage node;
the feedback module 9 is connected with the central control module 4 and used for sequencing according to the safety performance evaluation result and generating a storage reference table for feeding back the sequencing result of each storage node and the size of the available storage space of the corresponding storage node;
and the calling module 10 is connected with the central control module 4 and is used for utilizing the storage nodes according to the storage reference table.
The state information of the data storage node provided by the embodiment of the invention comprises the following steps: the load rate, the original performance value, the input/output read-write information, the input/output type abnormal information and other related information of the storage node are obtained.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention, and the scope of the present invention is not limited thereto, and any modification, equivalent replacement, and improvement made by those skilled in the art within the technical scope of the present invention disclosed herein, which is within the spirit and principle of the present invention, should be covered by the present invention.
Claims (10)
1. A big data-based storage node space capacity and security evaluation method is characterized by comprising the following steps:
acquiring the type, the number, the original data volume and other relevant basic information of storage nodes through a basic information acquisition module; acquiring relevant state data of the storage node through a state data acquisition module; screening currently available data storage nodes based on the acquired relevant state information of the storage nodes through a screening module;
secondly, controlling a safety coefficient calculation module by using a single chip microcomputer or a controller through a central control module to calculate the safety coefficient of each available data storage node; evaluating the performance of the storage node based on the acquired relevant state data of the storage node through a performance evaluation module;
thirdly, predicting and calculating the fault rate of each storage node through a fault rate prediction module; the safety performance of each storage node is evaluated through a safety evaluation module based on basic information, state data, safety factors, performance evaluation results and failure rate of the storage nodes;
sequencing according to the safety performance evaluation result through a feedback module, and generating a storage reference table for the sequencing result of each storage node and the size of the available storage space of the corresponding storage node to feed back; utilizing the storage nodes through a calling module according to the storage reference table;
in the second step, the calculating the safety factor of the storage node by the safety factor calculating module includes:
(1) acquiring defect parameters and risk parameters of storage nodes, and establishing a safety coefficient generation model based on a radial basis function neural network based on the acquired defect parameters and risk parameters of the storage nodes;
(2) determining the calculation weight of the defect parameters and the risk parameters in the safety coefficient generation model according to the historical defect parameters and the historical risk parameters;
(3) generating the safety coefficient of the storage node by utilizing the corresponding calculation weight of the defect parameter and the risk parameter and a pre-established safety coefficient generation model based on the radial basis function neural network;
in the first step, the specific process of the screening module screening the currently available data storage nodes based on the acquired relevant state information of the storage nodes is as follows:
establishing a corresponding data set according to the relevant state information of the storage nodes;
scanning data in the data set, calculating the distance between each data object and an adjacent object, accumulating to calculate the distance sum, and calculating the distance sum average value;
assuming that the sum of the distances of a certain data object is greater than the sum of the distances and the mean value, regarding the point as a storage node; moving the object from the data set to the storage node set, and repeating until all storage nodes are found;
in the second step, the evaluating the performance of the storage node by the performance evaluating module based on the acquired relevant state data of the storage node includes:
firstly, acquiring relevant state data of a storage node; calculating the residual performance value and the maximum occupied bandwidth value according to the load rate and the original performance value in the state data of each storage node;
then, obtaining the total residual performance value of the storage system according to the residual performance value of each unfilled storage node;
and finally, judging the performance of the storage nodes based on the performance allowance rate and the maximum occupied bandwidth value according to the ratio of the residual performance value of each storage node to the total residual performance value of all the storage nodes as the performance allowance rate.
2. The big-data-based storage node space capacity and safety assessment method according to claim 1, wherein in the step (1), the establishing of the safety coefficient generation model based on the radial basis function neural network based on the acquired defect parameters and risk parameters of the storage node comprises:
training the safety coefficient generation model based on the radial basis function neural network by using the historical defect parameters and the historical risk parameters; and correcting the trained safety coefficient generation model based on the radial basis function neural network by using the historical defect parameters and the historical risk parameters.
3. The big-data-based storage node space capacity and security evaluation method according to claim 1, wherein the maximum occupied bandwidth value calculation method comprises:
determining a calculation mode of occupied bandwidth of a storage node in a data read-write scene; based on the determined calculation mode of the occupied bandwidth of each node, estimating the occupied bandwidth of each node in a data reading and writing scene according to the original data quantity of the storage nodes; and determining the maximum occupied bandwidth in the occupied bandwidths of the nodes.
4. The big-data-based storage node space capacity and security evaluation method according to claim 1, wherein the judging the performance of the storage node based on the performance allowance rate and the maximum occupied bandwidth value comprises:
the larger the performance allowance rate is, the better the performance state is; a smaller maximum occupied bandwidth value represents a better performance state.
5. The big-data-based storage node space capacity and security assessment method according to claim 1, wherein the determining the performance of the storage node based on the performance allowance rate and the maximum occupied bandwidth value further comprises:
and carrying out homodromous processing on the performance allowance rate and the maximum occupied bandwidth value, respectively setting weight values of the performance allowance rate and the maximum occupied bandwidth value in the performance evaluation of the storage node, and evaluating the performance of the storage node based on the weight values and the obtained homodromous processing performance allowance rate and maximum occupied bandwidth value data.
6. The big-data-based storage node space capacity and security assessment method according to claim 1, wherein in step three, the pre-estimating and calculating the failure rate of each storage node by the failure rate pre-estimating module comprises:
determining an upper limit value of the read-write erasing use times of each storage node based on different storage node media types, and determining the fault probability of the storage nodes based on the obtained input/output read-write information, the input/output abnormal information and the upper limit value of different blocks of each storage node.
7. The method for evaluating space capacity and security of storage nodes based on big data as claimed in claim 1, wherein in step three, said evaluating the security performance of each storage node by the security evaluation module based on the basic information, status data, security factor, performance evaluation result, and failure rate of the storage node comprises:
1) basic information, state data, safety coefficients, performance evaluation results and fault rates of the storage nodes are used as safety evaluation indexes of the storage nodes, and corresponding weights are distributed to the safety evaluation indexes;
2) calculating the score of the safety evaluation index, and carrying out weighted average on the score of the safety evaluation index based on the weight of the safety evaluation index to obtain the score of the safety evaluation index;
3) and comparing the safety score with a preset safety evaluation standard to obtain an evaluation target safety evaluation conclusion.
8. A big-data based storage node space capacity and security assessment system implementing the big-data based storage node space capacity and security assessment method according to claims 1-7, wherein the big-data based storage node space capacity and security assessment system comprises:
the system comprises a basic information acquisition module, a state data acquisition module, a screening module, a central control module, a safety coefficient calculation module, a performance evaluation module, a fault rate estimation module, a safety evaluation module, a feedback module and a calling module;
the basic information acquisition module is connected with the central control module and is used for acquiring the type, the number, the original data volume and other related basic information of the storage nodes;
the state data acquisition module is connected with the central control module and used for acquiring the related state data of the storage nodes;
the screening module is connected with the central control module and used for screening available data storage nodes based on the acquired relevant state information of the storage nodes; the specific process of screening the currently available data storage node by the screening module based on the acquired relevant state information of the storage node is as follows: establishing a corresponding data set according to the relevant state information of the storage nodes; scanning data in the data set, calculating the distance between each data object and an adjacent object, accumulating to calculate the distance sum, and calculating the distance sum average value; assuming that the sum of the distances of a certain data object is greater than the sum of the distances and the mean value, regarding the point as a storage node; moving the object from the data set to the storage node set, and repeating until all storage nodes are found;
and the central control module is connected with the basic information acquisition module, the state data acquisition module, the screening module, the safety coefficient calculation module, the performance evaluation module, the fault rate estimation module, the safety evaluation module, the feedback module and the calling module and is used for controlling each module to normally work by utilizing a single chip microcomputer or a controller.
9. The big-data based storage node space capacity and security assessment system of claim 8, wherein the state information of the data storage node comprises:
the load rate, the original performance value, the input/output read-write information, the input/output type abnormal information and other related information of the storage node are obtained.
10. The big-data based storage node space capacity and security assessment system according to claim 8, wherein said big-data based storage node space capacity and security assessment system further comprises:
the safety coefficient calculation module is connected with the central control module and used for calculating the safety coefficient of each available data storage node; acquiring defect parameters and risk parameters of storage nodes, and establishing a safety coefficient generation model based on a radial basis function neural network based on the acquired defect parameters and risk parameters of the storage nodes; determining the calculation weight of the defect parameters and the risk parameters in the safety coefficient generation model according to the historical defect parameters and the historical risk parameters; generating the safety coefficient of the storage node by utilizing the corresponding calculation weight of the defect parameter and the risk parameter and a pre-established safety coefficient generation model based on the radial basis function neural network;
the performance evaluation module is connected with the central control module and used for evaluating the performance of the storage nodes based on the acquired relevant state data of the storage nodes;
the failure rate prediction module is connected with the central control module and used for predicting and calculating the failure rate of each storage node; determining an upper limit value of the read-write erasing use times of each storage node based on different storage node media types, and determining the fault probability of the storage nodes based on the obtained input and output read-write information, input and output abnormal information and the upper limit value of different blocks of each storage node;
the safety evaluation module is connected with the central control module and is used for evaluating the safety performance of each storage node based on the basic information, the state data, the safety coefficient, the performance evaluation result and the fault rate of the storage node; the safety performance evaluation module carries out safety performance evaluation on each storage node based on the basic information, the state data, the safety factor, the performance evaluation result and the failure rate of the storage node, and comprises the following steps: basic information, state data, safety coefficients, performance evaluation results and fault rates of the storage nodes are used as safety evaluation indexes of the storage nodes, and corresponding weights are distributed to the safety evaluation indexes; calculating the score of the safety evaluation index, and carrying out weighted average on the score of the safety evaluation index based on the weight of the safety evaluation index to obtain the score of the safety evaluation index; comparing the safety score with a preset safety evaluation standard to obtain an evaluated target safety evaluation conclusion;
the feedback module is connected with the central control module and used for sequencing according to the safety performance evaluation result and generating a storage reference table for feeding back the sequencing result of each storage node and the size of the available storage space of the corresponding storage node;
and the calling module is connected with the central control module and is used for utilizing the storage nodes according to the storage reference table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011491797.1A CN112416265A (en) | 2020-12-17 | 2020-12-17 | Storage node space capacity and security evaluation method and system based on big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011491797.1A CN112416265A (en) | 2020-12-17 | 2020-12-17 | Storage node space capacity and security evaluation method and system based on big data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112416265A true CN112416265A (en) | 2021-02-26 |
Family
ID=74775884
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011491797.1A Withdrawn CN112416265A (en) | 2020-12-17 | 2020-12-17 | Storage node space capacity and security evaluation method and system based on big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112416265A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118170325A (en) * | 2024-05-13 | 2024-06-11 | 邯郸鉴晨网络科技有限公司 | Big data storage and processing control method |
-
2020
- 2020-12-17 CN CN202011491797.1A patent/CN112416265A/en not_active Withdrawn
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118170325A (en) * | 2024-05-13 | 2024-06-11 | 邯郸鉴晨网络科技有限公司 | Big data storage and processing control method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108632365B (en) | Service resource adjusting method, related device and equipment | |
| CN104537076B (en) | A kind of file read/write method and device | |
| CN105007312A (en) | Method and system for controlling adaptive load-balancing of cloud computing server | |
| CN108923996B (en) | Capacity analysis method and device | |
| CN113010576A (en) | Method, device, equipment and storage medium for capacity evaluation of cloud computing system | |
| CN108156235A (en) | Online verification method, apparatus, computer equipment and storage medium | |
| CN114844791B (en) | Cloud service automatic management and distribution method and system based on big data and storage medium | |
| CN112348666A (en) | Method and device for determining system capacity | |
| CN112148779A (en) | Method, device and storage medium for determining service index | |
| CN112416265A (en) | Storage node space capacity and security evaluation method and system based on big data | |
| CN115525230A (en) | Storage resource allocation method and device, storage medium and electronic equipment | |
| CN117609888B (en) | Security risk evaluation method for cloud printing data transmission | |
| CN110084476B (en) | Case adjustment method, device, computer equipment and storage medium | |
| CN108228752B (en) | Data total export method, data export task allocation device and data export node device | |
| CN102546235A (en) | Performance diagnosis method and system of web-oriented application under cloud computing environment | |
| Radi et al. | Genetic-based virtual machines consolidation strategy with efficient energy consumption in cloud environment | |
| CN107277143A (en) | A kind of resource matched management method and device | |
| CN116820767A (en) | Cloud resource management method and device, electronic equipment and storage medium | |
| CN111401787A (en) | Site selection method, device and system | |
| CN110134575B (en) | Method and device for calculating service capacity of server cluster | |
| CN103593249A (en) | HA early warning method and virtual resource manager | |
| CN116703248B (en) | Data auditing method, device, electronic equipment and computer readable storage medium | |
| CN110740168A (en) | Self-adaptive method for multi-tenant server in cloud | |
| CN117972367B (en) | Data storage prediction method, data storage subsystem and intelligent computing platform | |
| CN111930299A (en) | Method for allocating memory units and related device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210226 |
|
| WW01 | Invention patent application withdrawn after publication |