CN114020717A - Method, device, equipment and medium for acquiring performance data of distributed storage system - Google Patents
Method, device, equipment and medium for acquiring performance data of distributed storage system Download PDFInfo
- Publication number
- CN114020717A CN114020717A CN202111199037.8A CN202111199037A CN114020717A CN 114020717 A CN114020717 A CN 114020717A CN 202111199037 A CN202111199037 A CN 202111199037A CN 114020717 A CN114020717 A CN 114020717A
- Authority
- CN
- China
- Prior art keywords
- information
- storage system
- metadata server
- data
- distributed file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 77
- 230000008569 process Effects 0.000 claims abstract description 26
- 230000008859 change Effects 0.000 claims description 35
- 238000004590 computer program Methods 0.000 claims description 12
- 238000005070 sampling Methods 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000012545 processing Methods 0.000 description 17
- 238000004458 analytical method Methods 0.000 description 7
- 238000004891 communication Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000007639 printing Methods 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011056 performance test Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application discloses a method and a device for acquiring performance data of a distributed file storage system, electronic equipment and a readable storage medium. The method comprises the steps of performing instrumentation on a target node position of a message processed by a metadata server in advance so as to collect operation semantic data; acquiring load information and operation semantic data information of a metadata server in the running process of a distributed file storage system; and automatically adjusting the cache data volume of the metadata server by analyzing the operation semantic data information and the load information. The method and the device can accurately and efficiently analyze the storage performance of the distributed file storage system.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method and an apparatus for acquiring performance data of a distributed file storage system, an electronic device, and a readable storage medium.
Background
With the wide application of computer technology, daily work and life can generate a large amount of computer data, and the storage technology is correspondingly developed, so that the distributed file system is applied. At present, a large amount of small files are generated in most application scenes of the distributed file system, and frequent operations such as reading, writing, deleting and the like are performed on the small files, which occupies a large amount of computing resources, so that the metadata service becomes a bottleneck of storage performance. When the distributed file storage performance influences the use of a user, on-site customer service personnel and research and development personnel are difficult to locate the bottleneck point in a short time due to the fact that the on-site environment is complex and the use scene of the user is not well known. Meanwhile, in the normal performance test process, the performance analysis efficiency is low due to the fact that the performance analysis is inconvenient, inaccurate and unintuitive.
In view of this, how to accurately and efficiently analyze the storage performance of the distributed file storage system is a technical problem to be solved by those skilled in the art.
Disclosure of Invention
The application provides a performance data acquisition method and device of a distributed file storage system, electronic equipment and a readable storage medium, which can accurately and efficiently analyze the storage performance of the distributed file storage system.
In order to solve the above technical problems, embodiments of the present invention provide the following technical solutions:
an embodiment of the present invention provides a method for acquiring performance data of a distributed file storage system, including:
performing instrumentation on a target node position of a message processed by a metadata server in advance so as to collect operation semantic data;
acquiring load information and operation semantic data information of a metadata server in the running process of a distributed file storage system;
and automatically adjusting the cache data volume of the metadata server by analyzing the operation semantic data information and the load information.
Optionally, after obtaining the load information and the operation semantic data information of the metadata server in the running process of the distributed file storage system, the method further includes:
determining a performance early warning condition in advance according to the operation times and/or time delay of the operation semantics;
determining the times and time delay information of each type of operation semantics according to the operation semantic data information;
judging whether the performance early warning condition is met or not according to the times and the time delay information of the current class of operation semantics for each class of operation semantics;
and if the performance early warning condition is met, generating alarm prompt information.
Optionally, before obtaining the load information and the operation semantic data information of the metadata server in the running process of the distributed file storage system, the method further includes:
a message queue is constructed in advance;
when the metadata server receives the operation semantics to be executed, recording and adding the operation semantics to be executed into the message queue;
and when the to-be-executed operation semantics are detected to be processed, deleting the to-be-executed operation semantics from the message queue.
Optionally, the cache data size of the metadata server is automatically adjusted by analyzing the operation semantic data information and the load information:
calculating the change information of the metadata server according to all operation semantics and each load according to a preset sampling interval frequency;
if the change information meets a cache increasing condition, increasing the cache data volume of the metadata server; and if the change information does not meet the cache increasing condition, reducing the cache data volume of the metadata server.
Optionally, the calculating the change information of the metadata server according to all the operation semantics and each load includes:
calling a change data calculation relation to calculate the change information, wherein the change data calculation relation is as follows:
Δ＝w1·W+w2·N+w3·Q+w4·L；
wherein Δ is the change information, w1W is the alarm weighting factor, W is the number of times of alarm prompt message generation, W2For the operation semantic weighting factor, N is the number of operation semantic changes, w3Is a message queue weighting factor, Q is the number of message changes in the message queue, w4Is the load weighting factor and L is the load change data.
Optionally, after obtaining the load information and the operation semantic data information of the metadata server in the running process of the distributed file storage system, the method further includes:
presetting a command interface;
and exporting the load information and the operation semantic data information to a target path through the command interface.
Another aspect of the embodiments of the present invention provides a performance data obtaining apparatus for a distributed file storage system, including:
the data acquisition module is used for carrying out instrumentation on the target node position of the message processed by the metadata server in advance so as to acquire operation semantic data;
the data acquisition module is used for acquiring load information and operation semantic data information of the metadata server in the operation process of the distributed file storage system;
and the cache adjusting module is used for automatically adjusting the cache data volume of the metadata server by analyzing the operation semantic data information and the load information.
Optionally, the system further comprises an alarm module;
the warning module is used for determining performance early warning conditions in advance according to the operation times and/or time delay of operation semantics; determining the times and time delay information of each type of operation semantics according to the operation semantic data information; judging whether the performance early warning condition is met or not according to the times and the time delay information of the current class of operation semantics for each class of operation semantics; and if the performance early warning condition is met, generating alarm prompt information.
An embodiment of the present invention further provides an electronic device, including a processor, where the processor is configured to implement the steps of the performance data obtaining method of the distributed file storage system according to any one of the preceding items when executing the computer program stored in the memory.
Finally, an embodiment of the present invention provides a readable storage medium, where a computer program is stored on the readable storage medium, and when the computer program is executed by a processor, the computer program implements the steps of the performance data obtaining method of the distributed file storage system according to any previous item.
The technical scheme provided by the application has the advantages that the operation semantic information is acquired by inserting the piles into the key nodes in the operation semantic processing process in the metadata server, and according to different scenes, the performance data of the metadata server can be better captured, so that the analysis of the performance bottleneck points of the operation semantics is facilitated. The current storage performance of the distributed file storage system can be accurately and efficiently determined according to the time consumption, times and other related information of operation semantics in the system operation process and the load information of the metadata server, the data amount cached by the metadata server is automatically adjusted according to the storage performance analysis result, the storage system of the distributed file storage system is optimized in time, the stability of the metadata server is maintained, and the maintainability of the distributed file storage system is improved.
In addition, the embodiment of the invention also provides a corresponding implementation device, electronic equipment and a readable storage medium for the performance data acquisition method of the distributed file storage system, so that the method has higher practicability, and the device, the electronic equipment and the readable storage medium have corresponding advantages.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the related art, the drawings required to be used in the description of the embodiments or the related art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a schematic flowchart of a method for acquiring performance data of a distributed file storage system according to an embodiment of the present invention;
FIG. 2 is a block diagram of an exemplary application scenario provided by an embodiment of the present invention;
fig. 3 is a structural diagram of a performance data acquiring apparatus of a distributed file storage system according to an embodiment of the present invention;
fig. 4 is a block diagram of an embodiment of an electronic device according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The terms "first," "second," "third," "fourth," and the like in the description and claims of this application and in the above-described drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "comprising" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements but may include other steps or elements not expressly listed.
Having described the technical solutions of the embodiments of the present invention, various non-limiting embodiments of the present application are described in detail below.
Referring to fig. 1, fig. 1 is a schematic flow chart of a method for acquiring performance data of a distributed file storage system according to an embodiment of the present invention, where the embodiment of the present invention may include the following:
s101: and performing instrumentation on the target node position of the message processed by the metadata server in advance so as to collect operation semantic data.
The target node in this embodiment is a node that takes a long time for the metadata server to process the message, and includes but is not limited to a message distribution node, a locking stage, a journal disk drop and other key nodes. The instrumentation is to modify the existing code during the code compiling period to obtain the evidence to generate a new code, and any method for implementing the program instrumentation can be adopted, such as a Java file mode or a byte code mode, which does not affect the implementation of the application. The operation semantics such as create, getattr, setattr, rename, delete, etc., i.e., normal operations of reading, writing, modifying, renaming, deleting, etc., are commands sent by the client to the metadata server. The method comprises the steps of performing instrumentation on key nodes in the operation semantics such as creation and deletion processing processes in a metadata server, and collecting key information such as operation semantic types, times, timestamps and the like as operation semantic data.
S102: and acquiring load information and operation semantic data information of a metadata server in the running process of the distributed file storage system.
The operation semantic data information of this embodiment may include operation semantic data information collected by looking up the state of each target node, including an operation semantic type, a duration of the operation semantic, and a number of times of the operation semantic. The load information refers to the relevant operation data information of each load of the metadata server.
S103: and automatically adjusting the cache data volume of the metadata server by analyzing the operation semantic data information and the load information.
And analyzing the operation semantic data information of the last step to obtain parameters such as times, maximum time, minimum time, average time and the like of each operation semantic. And calculating the load of the metadata server to obtain the cpu and the memory occupied by the metadata server. Performance bottleneck points of operation semantics can be obtained based on the data analysis results, and the data volume cached by the metadata server is automatically adjusted. In a fault scene of the file system, the data analysis results can feed back operation semantic information of processing failure or blocking to adjust the caching capacity of the metadata server. During or after the storage system runs, the time consumed by small file operation in the metadata server and information such as load and message length of the metadata server can be displayed in real time, and performance data of the metadata server can be captured well according to different scenes.
In the technical scheme provided by the embodiment of the invention, the operation semantic information is acquired by inserting the piles in the key nodes in the operation semantic processing process in the metadata server, so that the performance data of the metadata server can be better captured according to different scenes, and the analysis of the performance bottleneck point of the operation semantic is facilitated. The current storage performance of the distributed file storage system can be accurately and efficiently determined according to the time consumption, times and other related information of operation semantics in the system operation process and the load information of the metadata server, the data amount cached by the metadata server is automatically adjusted according to the storage performance analysis result, the storage system of the distributed file storage system is optimized in time, the stability of the metadata server is maintained, and the maintainability of the distributed file storage system is improved.
It should be noted that, in the present application, there is no strict sequential execution order among the steps, and as long as the logical order is met, the steps may be executed simultaneously or according to a certain preset order, and fig. 1 is only an exemplary manner, and does not represent that only the execution order is the order.
In the embodiment, after the key nodes in the operation semantic processing process are instrumented in the metadata server, in order to further improve the storage performance and improve the stability of the storage system, performance early warning conditions can be determined in advance according to the operation times and/or time delay of the operation semantics; determining the times and time delay information of each type of operation semantics according to the operation semantic data information; judging whether performance early warning conditions are met or not according to the times and time delay information of the current operation semantics for each type of operation semantics; and if the performance early warning condition is met, generating alarm prompt information.
In this embodiment, the performance early warning condition may be flexibly determined according to an actual application scenario, for example, a maximum time threshold and an average time threshold may be set according to the delay information, and if the current maximum time is greater than the maximum time threshold and the current average time is greater than the average time threshold, the performance early warning condition is satisfied. And processing the collected operation semantic data to obtain the times, the maximum time, the minimum time and the average time of each operation semantic, and generating an alarm when the time information of a certain operation semantic is greater than the maximum time threshold or the average time threshold.
In this embodiment, each received operation semantic message of the metadata server may be recorded and added to the message queue, and after a certain operation semantic processing is completed, the operation semantic message may be removed from the message queue, which may include the following contents:
a message queue is constructed in advance; when the metadata server receives the operation semantics to be executed, recording and adding the operation semantics to be executed into a message queue; and when the operation semantics to be executed are detected to be processed, deleting the operation semantics to be executed from the message queue.
Based on this, in the above embodiment, the key nodes in the operation semantic (creation, deletion) processing process in the metadata server are instrumented, message queue information and load information in the operation process of the storage system can be collected, then information such as time delay and times of operation semantics is collated, and finally the caching capacity of the metadata server is adjusted.
The foregoing embodiment does not limit how to execute S103, and this embodiment may also provide an optional implementation manner for automatically adjusting the cache data amount of the metadata server by analyzing the operation semantic data information and the load information, which may include:
calculating the change information of the metadata server according to all operation semantics and each load according to a preset sampling interval frequency; if the change information meets the cache increasing condition, increasing the cache data volume of the metadata server; and if the change information does not meet the cache increasing condition, reducing the cache data volume of the metadata server.
In this embodiment, a sampling interval, such as a sampling interval period or a sampling interval frequency, may be set in advance according to an actual application scenario, and at fixed time intervals, relevant information such as an increment of an operation semantic, an increment of a load, and the like is calculated, and then the cache capacity of the metadata server is automatically adjusted according to an analysis result, so that a performance bottleneck point of the metadata server may be better obtained in different application scenarios (including a failure).
As an alternative embodiment, the process of calculating the change information of the metadata server according to all the operation semantics and each load comprises:
calling the change data to calculate a relational expression to calculate the change information, wherein the relational expression of the change data calculation is as follows:
Δ＝w1·W+w2·N+w3·Q+w4·L；
wherein Δ is change information, w1W is the alarm weighting factor, W is the number of times of alarm prompt message generation, W2For the operation semantic weighting factor, N is the number of operation semantic changes, w3Is a message queue weighting factor, Q is the number of message changes in the message queue, w4Is the load weighting factor and L is the load change data. The alarm weighting coefficient, the operation semantic weighting coefficient, the load weighting coefficient and the message queue weighting coefficient can be flexibly selected according to actual application scenes.
In addition, in order to further improve the user experience, the application further provides that the information required by the user is printed by providing a command interface, and the method may include:
presetting a command interface; and exporting the load information and/or the operation semantic data information and/or the message queue information to a target path through a command interface, or directly printing out.
The embodiment of the invention also provides a corresponding device for the performance data acquisition method of the distributed file storage system, thereby further ensuring that the method has higher practicability. Wherein the means can be described separately from the functional module point of view and the hardware point of view. In the following, the performance data obtaining apparatus of the distributed file storage system according to the embodiments of the present invention is introduced, and the performance data obtaining apparatus of the distributed file storage system described below and the performance data obtaining method of the distributed file storage system described above may be referred to correspondingly.
Based on the angle of the functional module, referring to fig. 3, fig. 3 is a structural diagram of a performance data obtaining apparatus of a distributed file storage system according to an embodiment of the present invention, in a specific implementation manner, where the apparatus may include:
the data acquisition module 301 is configured to perform instrumentation on a target node position where the metadata server processes the message in advance, so as to acquire operation semantic data;
a data obtaining module 302, configured to obtain load information and operation semantic data information of a metadata server in an operation process of a distributed file storage system;
and the cache adjusting module 303 is configured to automatically adjust the cache data amount of the metadata server by analyzing the operation semantic data information and the load information.
Optionally, in some embodiments of this embodiment, the apparatus may include an alarm module; the warning module is used for determining performance early warning conditions in advance according to the operation times and/or time delay of the operation semantics; determining the times and time delay information of each type of operation semantics according to the operation semantic data information; judging whether performance early warning conditions are met or not according to the times and time delay information of the current operation semantics for each type of operation semantics; and if the performance early warning condition is met, generating alarm prompt information.
Optionally, in other embodiments of this embodiment, the apparatus may further include a message queue processing module, for example, configured to pre-construct a message queue; when the metadata server receives the operation semantics to be executed, recording and adding the operation semantics to be executed into a message queue; and when the operation semantics to be executed are detected to be processed, deleting the operation semantics to be executed from the message queue.
As an optional implementation manner of the foregoing embodiment, the apparatus may further include an information printing module, configured to export the load information, the operation semantic data information, and the message queue information to the target path through a preset command interface.
As an optional implementation manner of this embodiment, the cache adjusting module 303 may be configured to: calculating the change information of the metadata server according to all operation semantics and each load according to a preset sampling interval frequency; if the change information meets the cache increasing condition, increasing the cache data volume of the metadata server; and if the change information does not meet the cache increasing condition, reducing the cache data volume of the metadata server.
As an optional implementation manner of the foregoing embodiment, the cache adjusting module 303 may be further configured to: calling the change data to calculate a relational expression to calculate the change information, wherein the relational expression of the change data calculation is as follows:
Δ＝w1·W+w2·N+w3·Q+w4·L；
wherein Δ is change information, w1W is the alarm weighting factor, W is the number of times of alarm prompt message generation, W2For the operation semantic weighting factor, N is the number of operation semantic changes, w3Is a message queue weighting factor, Q is the number of message changes in the message queue, w4Is the load weighting factor and L is the load change data.
The functions of each functional module of the performance data acquiring device of the distributed file storage system according to the embodiments of the present invention may be specifically implemented according to the method in the above method embodiments, and the specific implementation process may refer to the description related to the above method embodiments, which is not described herein again.
Therefore, the storage performance of the distributed file storage system can be accurately and efficiently analyzed.
The performance data acquiring apparatus of the distributed file storage system mentioned above is described from the perspective of functional modules, and further, the present application also provides an electronic device described from the perspective of hardware. Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in fig. 4, the electronic device includes a memory 40 for storing a computer program; a processor 41, configured to execute a computer program to implement the steps of the performance data obtaining method of the distributed file storage system according to any of the above embodiments.
The processor 41 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the processor 41 may also be a controller, a microcontroller, a microprocessor or other data processing chip, and the like. The processor 41 may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 41 may also include a main processor and a coprocessor, where the main processor is a processor for Processing data in an awake state, and is also called a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 41 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content that the display screen needs to display. In some embodiments, processor 41 may further include an AI (Artificial Intelligence) processor for processing computational operations related to machine learning.
Memory 40 may include one or more computer-readable storage media, which may be non-transitory. Memory 40 may also include high speed random access memory as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. The memory 40 may in some embodiments be an internal storage unit of the electronic device, for example a hard disk of a server. The memory 40 may also be an external storage device of the electronic device in other embodiments, such as a plug-in hard disk provided on a server, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like. Further, the memory 40 may also include both an internal storage unit and an external storage device of the electronic device. The memory 40 can be used for storing application software installed in the electronic device and various data, such as: the code of the program that executes the vulnerability handling method, etc. may also be used to temporarily store data that has been output or is to be output. In this embodiment, the memory 40 is at least used for storing the computer program 401, wherein after being loaded and executed by the processor 41, the computer program can implement the relevant steps of the performance data obtaining method of the distributed file storage system disclosed in any one of the foregoing embodiments. In addition, the resources stored in the memory 40 may also include an operating system 402, data 403, and the like, and the storage manner may be a transient storage or a permanent storage. Operating system 402 may include, among other things, Windows, Unix, Linux, and the like. The data 403 may include, but is not limited to, data corresponding to performance data acquisition results of the distributed file storage system, and the like.
In some embodiments, the electronic device may further include a display 42, an input/output interface 43, a communication interface 44, alternatively referred to as a network interface, a power supply 45, and a communication bus 46. The display 42 and the input/output interface 43, such as a Keyboard (Keyboard), belong to a user interface, and the optional user interface may also include a standard wired interface, a wireless interface, and the like. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, as appropriate, is used for displaying information processed in the electronic device and for displaying a visualized user interface. The communication interface 44 may optionally include a wired interface and/or a wireless interface, such as a WI-FI interface, a bluetooth interface, etc., typically used to establish a communication connection between an electronic device and other electronic devices. The communication bus 46 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 4, but this does not indicate only one bus or one type of bus.
Those skilled in the art will appreciate that the configuration shown in fig. 4 is not intended to be limiting of the electronic device and may include more or fewer components than those shown, such as sensors 47, for example, to perform various functions.
The functions of the functional modules of the electronic device according to the embodiments of the present invention may be specifically implemented according to the method in the above method embodiments, and the specific implementation process may refer to the description related to the above method embodiments, which is not described herein again.
Therefore, the storage performance of the distributed file storage system can be accurately and efficiently analyzed.
It is to be understood that, if the performance data acquisition method of the distributed file storage system in the above embodiment is implemented in the form of a software functional unit and sold or used as a stand-alone product, it may be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application may be substantially or partially implemented in the form of a software product, which is stored in a storage medium and executes all or part of the steps of the methods of the embodiments of the present application, or all or part of the technical solutions. And the aforementioned storage medium includes: a U disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), an electrically erasable programmable ROM, a register, a hard disk, a multimedia card, a card type Memory (e.g., SD or DX Memory, etc.), a magnetic Memory, a removable magnetic disk, a CD-ROM, a magnetic or optical disk, and other various media capable of storing program codes.
Based on this, the embodiment of the present invention further provides a readable storage medium, which stores a computer program, and the computer program is executed by a processor, and the steps of the performance data obtaining method of the distributed file storage system according to any one of the above embodiments are provided.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. For hardware including devices and electronic equipment disclosed by the embodiment, the description is relatively simple because the hardware includes the devices and the electronic equipment correspond to the method disclosed by the embodiment, and the relevant points can be obtained by referring to the description of the method.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The method, the apparatus, the electronic device and the readable storage medium for acquiring performance data of the distributed file storage system provided by the present application are described in detail above. The principles and embodiments of the present invention are explained herein using specific examples, which are presented only to assist in understanding the method and its core concepts. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present application.
Claims (10)
1. A performance data acquisition method of a distributed file storage system is characterized by comprising the following steps:
performing instrumentation on a target node position of a message processed by a metadata server in advance so as to collect operation semantic data;
acquiring load information and operation semantic data information of a metadata server in the running process of a distributed file storage system;
and automatically adjusting the cache data volume of the metadata server by analyzing the operation semantic data information and the load information.
2. The method for acquiring performance data of a distributed file storage system according to claim 1, wherein after acquiring load information and operation semantic data information of a metadata server in the operation process of the distributed file storage system, the method further comprises:
determining a performance early warning condition in advance according to the operation times and/or time delay of the operation semantics;
determining the times and time delay information of each type of operation semantics according to the operation semantic data information;
judging whether the performance early warning condition is met or not according to the times and the time delay information of the current class of operation semantics for each class of operation semantics;
and if the performance early warning condition is met, generating alarm prompt information.
3. The method for acquiring performance data of a distributed file storage system according to claim 1, wherein before acquiring the load information and the operation semantic data information of the metadata server during the operation of the distributed file storage system, the method further comprises:
a message queue is constructed in advance;
when the metadata server receives the operation semantics to be executed, recording and adding the operation semantics to be executed into the message queue;
and when the to-be-executed operation semantics are detected to be processed, deleting the to-be-executed operation semantics from the message queue.
4. The method for acquiring performance data of a distributed file storage system according to any one of claims 1 to 3, wherein the amount of cache data of the metadata server is automatically adjusted by analyzing the operation semantic data information and the load information:
calculating the change information of the metadata server according to all operation semantics and each load according to a preset sampling interval frequency;
if the change information meets a cache increasing condition, increasing the cache data volume of the metadata server; and if the change information does not meet the cache increasing condition, reducing the cache data volume of the metadata server.
5. The method for acquiring performance data of a distributed file storage system according to claim 4, wherein the calculating the change information of the metadata server according to all operation semantics and each load comprises:
calling a change data calculation relation to calculate the change information, wherein the change data calculation relation is as follows:
Δ＝w1·W+w2·N+w3·Q+w4·L；
wherein Δ is the change information, w1W is the alarm weighting factor, W is the number of times of alarm prompt message generation, W2For the operation semantic weighting factor, N is the number of operation semantic changes, w3Is a message queue weighting factor, Q is the number of message changes in the message queue, w4Is the load weighting factor and L is the load change data.
6. The method for acquiring performance data of a distributed file storage system according to claim 5, wherein after acquiring the load information and the operation semantic data information of the metadata server during the operation of the distributed file storage system, the method further comprises:
presetting a command interface;
and exporting the load information and the operation semantic data information to a target path through the command interface.
7. A performance data acquisition apparatus of a distributed file storage system, comprising:
the data acquisition module is used for carrying out instrumentation on the target node position of the message processed by the metadata server in advance so as to acquire operation semantic data;
the data acquisition module is used for acquiring load information and operation semantic data information of the metadata server in the operation process of the distributed file storage system;
and the cache adjusting module is used for automatically adjusting the cache data volume of the metadata server by analyzing the operation semantic data information and the load information.
8. The apparatus for acquiring performance data of a distributed file storage system according to claim 7, further comprising an alarm module;
the warning module is used for determining performance early warning conditions in advance according to the operation times and/or time delay of operation semantics; determining the times and time delay information of each type of operation semantics according to the operation semantic data information; judging whether the performance early warning condition is met or not according to the times and the time delay information of the current class of operation semantics for each class of operation semantics; and if the performance early warning condition is met, generating alarm prompt information.
9. An electronic device comprising a processor and a memory, the processor being configured to carry out the steps of the performance data acquisition method of the distributed file storage system according to any one of claims 1 to 6 when executing a computer program stored in the memory.
10. A readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the performance data acquisition method of the distributed file storage system according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111199037.8A CN114020717A (en) | 2021-10-14 | 2021-10-14 | Method, device, equipment and medium for acquiring performance data of distributed storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111199037.8A CN114020717A (en) | 2021-10-14 | 2021-10-14 | Method, device, equipment and medium for acquiring performance data of distributed storage system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114020717A true CN114020717A (en) | 2022-02-08 |
Family
ID=80055993
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111199037.8A Pending CN114020717A (en) | 2021-10-14 | 2021-10-14 | Method, device, equipment and medium for acquiring performance data of distributed storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114020717A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117573907A (en) * | 2024-01-16 | 2024-02-20 | 北京航空航天大学杭州创新研究院 | Mobile robot data storage method and system |
-
2021
- 2021-10-14 CN CN202111199037.8A patent/CN114020717A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117573907A (en) * | 2024-01-16 | 2024-02-20 | 北京航空航天大学杭州创新研究院 | Mobile robot data storage method and system |
| CN117573907B (en) * | 2024-01-16 | 2024-04-26 | 北京航空航天大学杭州创新研究院 | Mobile robot data storage method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9448908B2 (en) | System and method for model based session management | |
| CN107480039B (en) | Small file read-write performance test method and device for distributed storage system | |
| CN108153587B (en) | Slow task reason detection method for big data platform | |
| US20140074452A1 (en) | System and method for automatic modeling of an application | |
| EP3159798B1 (en) | Method and system for determining user interface usage | |
| CN112506759B (en) | Automatic test method and device for servo system control software and storage medium | |
| CN110990053A (en) | Method for creating and using machine learning scheme template and device | |
| CN113268403B (en) | Time series analysis and prediction method, device, equipment and storage medium | |
| US20180143897A1 (en) | Determining idle testing periods | |
| CN101393535A (en) | Method and system for associating event and assembly while working | |
| CN113835643B (en) | Data storage method and device, electronic equipment and readable storage medium | |
| CN109543891A (en) | Method for building up, equipment and the computer readable storage medium of capacity prediction model | |
| CN111666201A (en) | Regression testing method, device, medium and electronic equipment | |
| CN114020717A (en) | Method, device, equipment and medium for acquiring performance data of distributed storage system | |
| CN109871205B (en) | Interface code adjustment method, device, computer device and storage medium | |
| CN114647548A (en) | Log generation method and device | |
| CN117235527A (en) | End-to-end containerized big data model construction method, device, equipment and medium | |
| CN116719579A (en) | AI model observability realization method and device, electronic equipment and storage medium | |
| CN113791860B (en) | Information conversion method, device and storage medium | |
| US9183388B2 (en) | Injustice detecting system, injustice detecting device and injustice detecting method | |
| CN115454770A (en) | Reporting method and device of embedded point information, storage medium and electronic equipment | |
| CN114706581A (en) | Image analysis method, image analysis device, computer equipment and storage medium | |
| CN114546793A (en) | Log generation method and device and computer readable storage medium | |
| CN111290895A (en) | Method, device, equipment and medium for testing IO performance of server | |
| CN113810342A (en) | Intrusion detection method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |