WO2017036428A2 - Procédé et dispositif de suggestion de changement de capacité - Google Patents

Procédé et dispositif de suggestion de changement de capacité Download PDF

Info

Publication number
WO2017036428A2
WO2017036428A2 PCT/CN2016/104851 CN2016104851W WO2017036428A2 WO 2017036428 A2 WO2017036428 A2 WO 2017036428A2 CN 2016104851 W CN2016104851 W CN 2016104851W WO 2017036428 A2 WO2017036428 A2 WO 2017036428A2
Authority
WO
WIPO (PCT)
Prior art keywords
resource pool
storage resource
data
storage
migration
Prior art date
Application number
PCT/CN2016/104851
Other languages
English (en)
Chinese (zh)
Other versions
WO2017036428A3 (fr
Inventor
黄升旗
韩银俊
陆平
赵培
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Publication of WO2017036428A2 publication Critical patent/WO2017036428A2/fr
Publication of WO2017036428A3 publication Critical patent/WO2017036428A3/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0629—Configuration or reconfiguration of storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/0647—Migration mechanisms

Definitions

  • the present invention relates to the field of storage, and in particular to a method and apparatus for recommending a capacity change.
  • the traditional hierarchical storage introduces a certain amount of time to the storage system by using a high-performance (read performance) storage medium with a certain proportion of space on the basis of the original massive low-performance storage medium.
  • the historical statistics are compared with the manually preset storage grading strategy, and the content stored on the low-level (low-performance) medium and the read access heat exceeds the set rising heat threshold is raised to the high-level (high-performance) storage medium.
  • Migration simultaneous down-migration to low-level (low-performance) storage media for content stored on high-level (high-performance) media with read access heat below a set thermal descent threshold.
  • the thermal data resides on the high-performance layer media as much as possible, and the cold data resides on the low-performance layer storage medium as much as possible.
  • the heat of the content changes with the access of the application, and becomes hot or changed when the statistical period is set.
  • the migration corresponding to cold or constant, hot and cold data is transparent to the application.
  • the biggest problem in the traditional way is that the migration strategy of the data fragmentation is manually preset.
  • the hierarchical structure of the resource pools at different levels is static, and the dynamic expansion/contraction of different hierarchical resource pools cannot be real-time perceived and synchronized. It is impossible to dynamically adapt to the range change of the heat threshold in the upper-layer application storage access model, which may result in waste that cannot be fully utilized by high-level (high-performance) storage media or frequent content migration and migration resulting in reduced access to actual services. .
  • the embodiment of the invention provides a method and a device for recommending capacity change, so as to at least solve the problem that the access performance of the migration policy is reduced by manual static setting.
  • a method for recommending a capacity change including: counting that the number of two-way migration between the first storage resource pool and the second storage resource pool on the data storage server cluster exceeds the pre-predetermined time period And a third data fragment of the migration number is set; and according to the data volume of the third data fragment, a capacity change suggestion for the first storage resource pool is reported.
  • the method further includes: reporting alarm information for the third data fragment, where the alarm information includes: an alarm used to indicate that the current migration policy performance is low, and/or And an alarm indicating that the third data fragment is frequently migrated.
  • the method further includes: Activating a timer; if the counter times out and does not receive the response message of the capacity change recommendation, raising a first access heat threshold, and/or raising a second access heat threshold, wherein the first access The heat threshold is a minimum access heat value of the first data fragment that needs to be migrated from the second storage resource pool to the first storage resource pool, and the second access heat threshold is required by the first storage resource pool. The maximum access heat value of the second data fragment migrated to the second storage resource pool.
  • a capacity change suggesting apparatus comprising: a third data fragmentation statistics module, configured to count the first storage resource pool and the first on the data storage server cluster within a preset time period The second data fragment of the second storage resource pool exceeds the preset number of migrations; the capacity change recommendation reporting module is configured to report the first storage resource pool according to the data volume of the third data fragment. Capacity change recommendations.
  • the device further includes: an alarm information reporting module, configured to report alarm information for the third data fragment, wherein the alarm information includes: an alarm used to indicate that the performance of the current migration policy is low, and / or an alarm for indicating that the third data fragment is frequently migrated.
  • an alarm information reporting module configured to report alarm information for the third data fragment, wherein the alarm information includes: an alarm used to indicate that the performance of the current migration policy is low, and / or an alarm for indicating that the third data fragment is frequently migrated.
  • the apparatus further comprises: a timer starting module, configured to start a timer; a first access heat threshold and/or a second access heat threshold boosting module, configured to time out the counter and not receive the capacity
  • a timer starting module configured to start a timer
  • a first access heat threshold and/or a second access heat threshold boosting module configured to time out the counter and not receive the capacity
  • the first access heat threshold is raised, and/or the second access heat threshold is raised, wherein the first access heat threshold needs to be migrated from the second storage resource pool to the a minimum access heat value of the first data fragment of the first storage resource pool, where the second access heat threshold is a second data fragment that needs to be migrated from the first storage resource pool to the second storage resource pool Maximum access heat value.
  • a computer storage medium is further provided, and the computer storage medium may store an execution instruction for performing the implementation of the capacity change suggestion method in the foregoing embodiment.
  • the third data fragment whose number of bidirectional migrations between the first storage resource pool and the second storage resource pool on the data storage server cluster exceeds the preset migration number is counted in the preset time period;
  • the data volume of the data fragment is reported in the manner of the capacity change of the first storage resource pool, which solves the problem that the access performance of the migration policy is reduced due to manual static setting, and improves the access performance of the service.
  • FIG. 1 is a flowchart of a migration policy adjustment method according to an embodiment of the present invention.
  • FIG. 2 is a flow chart of a method for recommending a capacity change according to an embodiment of the present invention
  • FIG. 3 is a schematic structural diagram of a migration policy adjustment apparatus according to an embodiment of the present invention.
  • FIG. 4 is a first schematic structural diagram 1 of a migration policy adjustment apparatus according to an embodiment of the present invention.
  • FIG. 5 is a second schematic structural diagram of a migration policy adjustment apparatus according to an embodiment of the present invention.
  • FIG. 6 is a third schematic structural diagram of a migration policy adjustment apparatus according to an embodiment of the present invention.
  • FIG. 7 is a fourth schematic structural diagram of a migration policy adjusting apparatus according to an embodiment of the present invention.
  • FIG. 8 is a schematic diagram 5 of a preferred structure of a migration policy adjustment apparatus according to an embodiment of the present invention.
  • FIG. 9 is a schematic structural diagram of a capacity change suggesting apparatus according to an embodiment of the present invention.
  • FIG. 10 is a first schematic structural diagram 1 of a capacity change suggesting apparatus according to an embodiment of the present invention.
  • FIG. 11 is a second schematic structural diagram of a capacity change suggesting apparatus according to an embodiment of the present invention.
  • FIG. 12 is a schematic structural diagram of a smart hierarchical storage system according to a preferred embodiment of the present invention.
  • FIG. 13 is a flow chart of a smart hierarchical storage method in accordance with a preferred embodiment of the present invention.
  • FIG. 1 is a flowchart of a migration policy adjustment method according to an embodiment of the present invention. As shown in FIG. 1, the process includes the following steps:
  • Step S102 collecting the capacity information of the first storage resource pool on the data storage server cluster in the preset time period, where the first storage resource pool is used to store the data fragment whose access heat value exceeds the preset access heat threshold;
  • Step S104 Determine, according to the capacity information, whether a storage capacity of the first storage resource pool changes.
  • step S106 when it is determined that the storage capacity changes, the migration strategy of the data fragment of the data storage server cluster is adjusted.
  • the migration policy of the data fragment is adjusted according to the change of the storage capacity of the first storage resource pool; for example, in the case that the first storage resource pool is a high-performance storage resource pool, the foregoing steps may be dynamically
  • the change of the storage capacity of the high-performance storage resource pool adjusts the migration policy.
  • the high-performance storage medium waste and service access performance caused by manual static setting of the migration policy are solved. Reduced problems can improve the utilization of high-performance storage media and improve service access performance.
  • the method further includes: counting access information of the data fragments on the metadata server cluster in the preset time period; and collecting, according to the access information, the access heat value of the data fragments in the preset time period; The value determines the data fragment to be migrated in the data fragment, and the data fragment to be migrated is migrated.
  • the above-mentioned access heat value refers to the frequency at which data fragments are accessed. The higher the frequency of access, the higher the access heat value.
  • the specific access heat value statistics mode can be designed according to actual needs, and the statistical mode is not limited in the embodiment of the present invention.
  • Data fragmentation is generally stored in multiple In a storage resource pool, for example, according to storage performance classification, it can be divided into a high-performance storage resource pool (such as a storage resource pool of SSD media) and a low-performance storage resource pool (for example, a Serial Advanced Technology Attachment (Serial Advanced Technology Attachment, Referred to as SATA)).
  • the data fragment stored in the high-performance storage resource pool is a data fragment with a large access value. Because these data fragments are accessed frequently, the read performance is high; otherwise, the low-performance storage resource pool is stored.
  • the data fragmentation is to access data fragments with lower heat values.
  • the remaining storage space of the first storage resource pool (for example, a high performance storage resource pool) may be used. And the amount of data to be migrated to the first storage resource pool is processed.
  • the first storage resource pool (eg, the low performance storage resource pool) needs to be migrated to the first storage resource pool according to the access heat value.
  • a first data amount of the data fragment determining whether the remaining storage space of the first storage resource pool satisfies the storage requirement of the first data fragment; and determining that the remaining storage space of the first storage resource pool satisfies the first data fragment
  • the first data fragment is migrated to the first storage resource pool.
  • the storage resource pool determines, according to the remaining storage space of the first storage resource pool and the first data amount, that the first needs to be determined by the first And storing, by the storage resource pool, a second data quantity of the second data fragment of the second storage resource pool, where the second data quantity is greater than or equal to a difference between the first data quantity and the remaining storage space;
  • the two data fragments are migrated to the second storage resource pool; the first data fragment of the first data amount is migrated to the first storage resource pool.
  • the minimum data volume migration between the high performance storage resource pool and the low performance storage resource pool is implemented, thereby avoiding data.
  • a large number of shards are migrated.
  • the utilization of the high-performance storage resource pool is also improved, and the access performance of the data shards is improved.
  • step S106 when the migration policy is adjusted, the migration information of the data fragment in the preset time period may be counted; and according to the migration information, the first storage resource pool and the second storage resource pool are determined to be in the preset time period.
  • the fragment resides in the first storage resource pool; or determines that the number of bidirectional migrations between the first storage resource pool and the second storage resource pool in the preset time period is lower than the fourth data fragment of the preset migration number;
  • the fourth data fragment resides in the second storage resource pool within a preset multiple access popularity value statistics period after the current access heat value statistics period.
  • the migration policy of adjusting the data fragment of the data storage server cluster may further include at least one of: determining the first storage When the capacity of the resource pool is increased, the first access heat threshold is lowered; when it is determined that the capacity of the first storage resource pool is increased, the second access heat threshold is decreased; and when it is determined that the capacity of the first storage resource pool is decreased, Increase the first access heat threshold And determining, in the case that the first storage resource pool is reduced in capacity, the second access heat threshold is increased; wherein the first access heat threshold is a first data point that needs to be migrated from the second storage resource pool to the first storage resource pool.
  • the minimum access heat value of the slice, and the second access heat threshold is a maximum access heat value of the second data slice that needs to be migrated from the first storage resource pool to the second storage resource pool.
  • the access heat threshold of the uplink/downward migration can be adjusted according to the capacity change of the high-performance storage resource pool, so that the data volume of the hot data fragment can be expanded according to the capacity change of the high-performance storage resource pool or shrink.
  • the utilization rate of the first storage resource pool on the data storage service cluster may be determined according to the capacity information.
  • the utilization rate is increased.
  • the access heat value statistics period is increased, and/or the second is decreased.
  • the access popularity threshold wherein the second access heat threshold is a maximum access heat value of the second data fragment that needs to be migrated from the first storage resource pool to the second storage resource pool.
  • the method further includes: counting a third data fragment whose number of bidirectional migrations between the first storage resource pool and the second storage resource pool exceeds a preset migration number within a preset time period; reporting the third data fragmentation
  • the alarm information includes: an alarm for indicating that the performance of the current migration policy is low, and/or an alarm for indicating that the third data fragment is frequently migrated.
  • the capacity change suggestion for the first storage resource pool may be reported according to the data amount of the third data fragment.
  • the high-level follow-up can adjust the capacity of the high-performance storage resource pool according to the capacity change proposal.
  • the timer may also be started; if the counter expires and the response message of the capacity change recommendation is not received, the first access heat threshold is raised, and/or And raising a second access heat threshold, where the first access heat threshold is a minimum access heat value of the first data fragment that needs to be migrated from the second storage resource pool to the first storage resource pool, and the second access heat threshold is required by The maximum access heat value of the second data fragment of the first storage resource pool migrated to the second storage resource pool.
  • the third data fragment can be frequently migrated by adjusting the access heat threshold.
  • first data fragment “first data fragment”, “second data fragmentation”, “third data fragmentation” or “fourth data fragmentation” referred to in the embodiment of the present invention may be one. Or multiple, and, in general, due to the large amount of data in the storage system, the migrated data is generally also a data slice set composed of a plurality of data fragments.
  • FIG. 2 is a flowchart of a capacity change suggestion method according to an embodiment of the present invention. As shown in FIG. 2, the process includes the following steps:
  • Step S202 the third data fragment whose number of bidirectional migrations between the first storage resource pool and the second storage resource pool on the data storage server cluster exceeds the preset migration number in the preset time period is counted;
  • Step S204 Report a capacity change suggestion for the first storage resource pool according to the data volume of the third data fragment.
  • the capacity change proposal of the first storage resource pool (for example, a high-performance storage resource pool) may be reported according to the migration of the data fragment, so that the upper layer can automatically recommend the capacity of the first storage resource pool according to the capacity change proposal.
  • the problem is that the high-performance storage medium is wasted and the access performance of the service is reduced due to the static setting of the migration policy.
  • the utilization of the high-performance storage medium can be improved and the access performance of the service can be improved.
  • the foregoing may also be reported to The alarm information of the three data fragments, wherein the alarm information includes: an alarm for indicating that the performance of the current migration policy is low, and/or an alarm for indicating that the third data fragment is frequently migrated.
  • the timer may be started; if the counter expires and the response message of the capacity change recommendation is not received, the first access heat threshold is raised, and/or, The second access heat threshold is increased, where the first access heat threshold is a minimum access heat value of the first data fragment that needs to be migrated from the second storage resource pool to the first storage resource pool, and the second access heat threshold is required to be The maximum access heat value of the second data fragment of the storage resource pool migrated to the second storage resource pool.
  • the third data fragment can be frequently migrated by adjusting the access heat threshold.
  • first data fragment “first data fragment”, “second data fragmentation”, “third data fragmentation” or “fourth data fragmentation” referred to in the embodiment of the present invention may be one. Or multiple, and, in general, due to the large amount of data in the storage system, the migrated data is generally also a data slice set composed of a plurality of data fragments.
  • the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware, but in many cases, the former is A better implementation.
  • the technical solution of the present invention which is essential or contributes to the prior art, may be embodied in the form of a software product stored in a storage medium (such as ROM/RAM, disk,
  • the optical disc includes a number of instructions for causing a terminal device (which may be a cell phone, a computer, a server, or a network device, etc.) to perform the methods of various embodiments of the present invention.
  • a migration policy adjustment apparatus is further provided to implement the foregoing embodiments and preferred implementation manners, which are not described again.
  • the term "module” may implement a combination of software and/or hardware of a predetermined function.
  • the apparatus described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware, is also possible and contemplated.
  • FIG. 3 is a schematic structural diagram of a migration policy adjustment apparatus according to an embodiment of the present invention.
  • the apparatus includes: a capacity information statistics module 32, a capacity information determination module 34, and a migration policy adjustment module 36, wherein capacity information is counted.
  • the module 32 is configured to collect the capacity information of the first storage resource pool on the data storage server cluster in the preset time period, where the first storage resource pool is used to store the data fragment whose access heat value exceeds the preset access heat threshold.
  • the capacity information judging module 34 is configured to determine whether the storage capacity of the first storage resource pool changes according to the capacity information; and the migration policy adjustment module 36 is configured to adjust the data storage server cluster when it is determined that the storage capacity changes.
  • the data Fragmentation migration strategy is configured to collect the capacity information of the first storage resource pool on the data storage server cluster in the preset time period, where the first storage resource pool is used to store the data fragment whose access heat value exceeds the preset access heat threshold.
  • the capacity information judging module 34 is configured to
  • FIG. 4 is a first schematic structural diagram of a migration policy adjustment apparatus according to an embodiment of the present invention.
  • the capacity information statistics module 32 is further configured to: count statistics on a cluster of metadata servers within a preset time period.
  • the access information of the data fragmentation device includes: an access heat value statistics module 42 coupled to the capacity information statistics module 32, configured to count the access heat value of the data fragments in the preset time period according to the access information;
  • the migration module 44 is coupled to the access heat value statistics module 42 and configured to determine data fragments to be migrated in the data fragment according to the access heat value, and migrate the data fragments to be migrated.
  • the data fragment migration module 44 includes: a first data amount determining unit 442, configured to determine, according to the access heat value, that the second storage is required, if the occupied ratio of the first storage resource pool does not reach a preset full threshold.
  • the resource pool is migrated to the first data volume of the first data fragment of the first storage resource pool;
  • the storage requirement determining unit 444 is coupled to the first data amount determining unit 442, and configured to determine whether the remaining storage space of the first storage resource pool is Satisfying the storage requirement of the first data fragment;
  • the first data fragment migration unit 446 is coupled to the storage requirement determining unit 444, and configured to determine that the remaining storage space of the first storage resource pool satisfies the storage requirement of the first data fragment In the case of the first data fragment, the first data fragment is migrated to the first storage resource pool.
  • the data fragment migration module 44 further includes: a second data amount determining unit 448 coupled to the storage requirement determining unit 444, configured to determine that the remaining storage space of the first storage resource pool cannot satisfy the first data fragmentation
  • the second data amount of the second data fragment that needs to be migrated from the first storage resource pool to the second storage resource pool is determined according to the remaining storage space of the first storage resource pool and the first data amount, where The second data amount is greater than or equal to the difference between the first data amount and the remaining storage space
  • the second data fragment migration unit 449 is coupled to the second data amount determining unit 448 and configured to set the second data amount of the second data.
  • the fragment is migrated to the second storage resource pool; wherein, the first data fragment migration unit 446 is further configured to: after migrating the second data fragment of the second data amount to the second storage resource pool, the first data volume The first data fragment is migrated to the first storage resource pool.
  • the migration policy adjustment module 36 includes: a migration information statistics unit 362 configured to count migration information of data fragments in a preset time period; a third data fragment determination unit 364 coupled to the migration information statistics unit 362, configured And determining, according to the migration information, a third data fragment whose number of bidirectional migrations between the first storage resource pool and the second storage resource pool exceeds a preset migration number within a preset time period; and/or, the fourth data fragmentation is determined
  • the unit 366 is coupled to the migration information statistics unit 362, and configured to determine, according to the migration information, that the number of bidirectional migrations between the first storage resource pool and the second storage resource pool is lower than the preset migration number in the preset time period.
  • a third data fragment resident unit 365 coupled to the third data fragment determining unit 364, configured to preset a plurality of accesses after the current access heat value statistical period in the case of determining that the storage capacity is increased
  • the third data fragment resides in the first storage resource pool during the heat value statistics period; and/or the fourth data fragment resident unit 367 is coupled to the fourth data point
  • the slice determining unit 366 is configured to: in the case that it is determined that the capacity of the first storage resource pool is reduced, the fourth data fragment resides in the preset multiple access heat value statistics period after the current access heat value statistics period Two storage resource pools.
  • the migration policy adjustment module 36 further includes at least one of the following: the first access heat threshold reduction unit is configured to reduce the first access heat threshold in the case that the first storage resource pool capacity is determined to be increased; the second access heat a threshold reduction unit, configured to decrease a second access heat threshold when it is determined that the capacity of the first storage resource pool increases; An access heat threshold raising unit is configured to: when it is determined that the capacity of the first storage resource pool is reduced, the first access heat threshold is increased; and the second access heat threshold increasing unit is configured to determine that the first storage resource pool is reduced in capacity.
  • the first access heat threshold reduction unit is configured to reduce the first access heat threshold in the case that the first storage resource pool capacity is determined to be increased
  • the second access heat a threshold reduction unit configured to decrease a second access heat threshold when it is determined that the capacity of the first storage resource pool increases
  • An access heat threshold raising unit is configured to: when it is determined that the capacity of the first storage resource pool is reduced, the first access heat threshold is increased
  • the second access heat threshold increasing unit is configured to determine that
  • the second access heat threshold is raised, where the first access heat threshold is a minimum access heat value of the first data fragment that needs to be migrated from the second storage resource pool to the first storage resource pool, and the second access heat threshold is used.
  • FIG. 5 is a schematic diagram of a preferred structure of a migration policy adjustment apparatus according to an embodiment of the present invention.
  • the apparatus further includes: a utilization determination module 52 coupled to the capacity information statistics module 32, configured to determine Whether the utilization of the first storage resource pool in the continuous multiple access popularity value statistics period is lower than the preset utilization rate; the access heat value statistics period and/or the second access heat threshold adjustment module 54 is coupled to the utilization determination module 52.
  • the access popularity threshold is a maximum access heat value of the second data fragment that needs to be migrated from the first storage resource pool to the second storage resource pool.
  • FIG. 6 is a schematic diagram of a preferred structure of a migration policy adjustment apparatus according to an embodiment of the present invention.
  • the capacity information statistics module 32 is further configured to: count the first storage resource pool in a preset time period and The third data fragment of the second storage resource pool exceeds the preset number of migrations.
  • the device further includes: an alarm information reporting module 62 coupled to the capacity information statistics module 32, configured to report the third data fragmentation.
  • the alarm information includes: an alarm for indicating that the performance of the current migration policy is low, and/or an alarm for indicating that the third data fragment is frequently migrated.
  • FIG. 7 is a schematic diagram of a preferred structure of a migration policy adjustment apparatus according to an embodiment of the present invention.
  • the apparatus further includes: a capacity change suggestion reporting module 72 coupled to the capacity information statistics module 32, configured to be configured according to The amount of data of the third data fragment is reported to suggest a capacity change of the first storage resource pool.
  • FIG. 8 is a schematic diagram of a preferred structure of a migration policy adjustment apparatus according to an embodiment of the present invention. As shown in FIG. 8, the apparatus further includes: a timer activation module 82 coupled to the capacity change suggestion reporting module 72, configured to be activated.
  • a timer a timer; a first access heat threshold and/or a second access heat threshold boost module 84 coupled to the timer activation module 82, configured to raise the first in the event that the counter times out and does not receive a response message to the capacity change recommendation Accessing the popularity threshold, and/or increasing the second access heat threshold, wherein the first access heat threshold is a minimum access heat value of the first data fragment that needs to be migrated from the second storage resource pool to the first storage resource pool,
  • the second access heat threshold is a maximum access heat value of the second data fragment that needs to be migrated from the first storage resource pool to the second storage resource pool.
  • the embodiment also provides a capacity change suggesting device for implementing the above-mentioned embodiments and preferred embodiments, which have not been described again.
  • FIG. 9 is a schematic structural diagram of a capacity change suggesting apparatus according to an embodiment of the present invention.
  • the apparatus includes: a third data fragmentation statistics module 92 and a capacity change suggestion reporting module 94, wherein the third data fragmentation
  • the statistics module 92 is configured to collect a third data fragment whose number of bidirectional migrations between the first storage resource pool and the second storage resource pool exceeds a preset migration number in the data storage server cluster within a preset time period;
  • the module 94 is coupled to the third data fragmentation statistics module 92, and is configured to report a capacity change suggestion for the first storage resource pool according to the data volume of the third data fragment.
  • FIG. 10 is a schematic diagram of a preferred structure of a capacity change suggesting apparatus according to an embodiment of the present invention.
  • the apparatus further includes: an alarm information reporting module 102 coupled to the third data fragmentation counting module 92, and configured.
  • the alarm information includes: an alarm for indicating that the performance of the current migration policy is low, and/or an alarm for indicating that the third data fragment is frequently migrated.
  • FIG. 11 is a schematic diagram of a preferred structure of a capacity change suggesting apparatus according to an embodiment of the present invention. As shown in FIG. 11, the apparatus further includes: a timer starting module 112 coupled to the capacity change suggestion reporting module 94, configured to be activated.
  • a timer a timer; a first access heat threshold and/or a second access heat threshold boost module 114, coupled to the timer activation module 112, configured to raise the first in the event that the counter times out and does not receive a response message to the capacity change recommendation Accessing the popularity threshold, and/or increasing the second access heat threshold, wherein the first access heat threshold is a minimum access heat value of the first data fragment that needs to be migrated from the second storage resource pool to the first storage resource pool,
  • the second access heat threshold is a maximum access heat value of the second data fragment that needs to be migrated from the first storage resource pool to the second storage resource pool.
  • each of the above modules may be implemented by software or hardware.
  • the foregoing may be implemented by, but not limited to, the foregoing modules are all located in the same processor; or, the modules are located in multiple In the processor.
  • Embodiments of the present invention also provide a software for performing the technical solutions described in the above embodiments and preferred embodiments.
  • Embodiments of the present invention also provide a storage medium.
  • the above storage medium may be configured to store program code for performing the following steps:
  • Step S102 collecting the capacity information of the first storage resource pool on the data storage server cluster in the preset time period, where the first storage resource pool is used to store the data fragment whose access heat value exceeds the preset access heat threshold;
  • Step S104 Determine, according to the capacity information, whether a storage capacity of the first storage resource pool changes.
  • step S106 when it is determined that the storage capacity changes, the migration strategy of the data fragment of the data storage server cluster is adjusted.
  • Embodiments of the present invention also provide a storage medium.
  • the above storage medium may be configured to store program code for performing the following steps:
  • Step S202 the third data fragment whose number of bidirectional migrations between the first storage resource pool and the second storage resource pool on the data storage server cluster exceeds the preset migration number in the preset time period is counted;
  • Step S204 Report a capacity change suggestion for the first storage resource pool according to the data volume of the third data fragment.
  • the foregoing storage medium may include, but is not limited to, a USB flash drive, a Read-Only Memory (ROM), and a Random Access Memory (RAM).
  • ROM Read-Only Memory
  • RAM Random Access Memory
  • the hardware resource adjustment suggestions of the intelligent resource pool are given, and the hardware configuration is adapted to the service application performance requirement; or the intelligent dynamic grading strategy adjustment is performed to reduce/eliminate the internal migration jitter and optimize the system access performance.
  • a preferred embodiment of the present invention will be described and illustrated by taking a mass file storage application scenario based on a distributed file system and a Write Once & Read Mostly (WORM) model in the cloud storage domain as an example.
  • a high-level storage resource pool (or a high-performance storage resource pool) is equivalent to the foregoing first storage resource pool; and a low-level storage resource pool (or a low-performance storage resource pool) is equivalent to the second Storage resource pool.
  • an intelligent grading engine embedded in a distributed file storage system, and the intelligent grading engine can sense the dynamic change of the current service hotspot model in real time (the spread and shrinkage of the hot content range) and different levels.
  • Resource pool expansion/contraction dynamic change real-time statistical analysis of data migration direction and data volume for the current period of time (including all data fragmentation heat statistics period and migration period in multiple periods), service access distribution and hierarchical performance improvement benefit statistics
  • Intelligent adaptive adjustment of hierarchical policies such as content rise/fall heat threshold and corresponding migration content in hierarchical storage, and intelligent hardware configuration adjustment suggestions for system administrators for current business hotspot access models; The system optimizes the performance matching of the current business model and optimizes the utilization of the storage system performance.
  • the solution of the preferred embodiment of the present invention includes the following three parts:
  • the first part is the real-time calculation of data migration cost and performance improvement benefit under the current hierarchical configuration of the system:
  • the data migration (two-way) is recorded in the real-time system.
  • the data migration takes precedence over the different levels of storage media groups between the single nodes in the storage server cluster.
  • the records can include: migration direction, migration start time, migration end time, and migration occupation.
  • the second part the business access hotspot model perception and hierarchical strategy intelligent adaptive adjustment:
  • the access type, the access amount, and the access frequency of the stored data fragments are recorded in real time on the data fragmentation unit in the storage server cluster, and the current system is calculated and recorded in the current system.
  • the access to the lower-level storage resource pool is high for the case where the overall occupancy ratio of the upper-level storage resource pool does not reach the full threshold (assuming the capacity corresponding to the full threshold is T1).
  • the migration amount equal to the data fragment content in the high-level storage resource pool (assuming the total amount of migration required is A1), when The ascending migration plus the current data capacity that does not need to be migrated (assumed to be C1) exceeds the high-level storage pool full threshold (ie: C1+A1>T1), first of all, the access heat in the high-level storage resource pool is lower than or equal to low.
  • the upward migration (A1) is upgraded.
  • the space occupancy statistics of different hierarchical storage resource pools (especially high-level storage resource pools) in each cycle are simultaneously increased, when hotspot ranges occur.
  • Shrinking causes the high-level storage resource pool utilization to be below the set maximum value of the full threshold for a continuous period of time (for example, multiple heat statistics periods): 1) automatically adjust the heat statistics period, for example, depending on the hot content The degree of shrinkage can adjust the statistical period to 1.5 or more times, reducing the unnecessary migration consumption in different levels of storage resource pools; 2) automatically lowering the data fragmentation without exceeding the full threshold of the high-level storage resource pool
  • the thermal threshold of the ascending migration ensures a moderate fullness and high utilization of the high-level storage resource pool;
  • the response is divided into two priorities.
  • the data fragmentation content that is frequently bidirectionally migrated in consecutive cycles can stay at the high level (in the case of high-level storage resource pool expansion)/low level (in the high-level storage resource pool) for a certain period of time. In the case of volume reduction) in the storage resource pool.
  • the high-level storage resource pool cannot reach the full threshold (assuming the full threshold corresponds to the capacity T1).
  • the hotspot data in the previous statistical period in the high-level level is continuously replaced by the hotspot data in the new statistical period in the lower level, thereby Two-way frequent data migration between two levels of storage resource pools occurs. This migration will result in a reduction in the system's external service capabilities.
  • the minimum mode bearer refers to the current storage redundancy mode in the system, such as multiple copy or Erasure Coding (EC) mode, and the storage service cluster meets the data volume of the ascending migration (A1) in the high-level storage.
  • the minimum size at which the resource pool resides. For example, in n(n> 1), only one hotspot copy resides in the high-level storage resource pool, and the remaining n-1 copies still reside in the low-level storage resource pool; in EC mode (assumed redundancy)
  • the ratio of n:m) only n copies of data reside in the high-level storage resource pool, and the remaining m copies of data reside in the low-level storage resource pool.
  • the intelligent classification engine gives low performance penalty and frequent migration to the data fragments of frequent bidirectional migration according to the recorded multiple heat statistics and the data fragment migration information and the access heat statistics information in the data migration period. police.
  • high-level storage resource pool expansion proposals according to the number of frequently migrated data fragments, for example, minimum (minimum bearer mode) capacity and optimal capacity two quantity value suggestions.
  • the low-level storage resource pool data fragmentation content rising migration heat threshold and the high-level storage resource pool data fragmentation content drop migration heat threshold may be the same value.
  • the following describes the internal adaptation and adjustment process of intelligent tiered storage by taking the expansion of the storage resource pool and the application hotspot model change (hotspot extension) as an example.
  • FIG. 12 is a schematic structural diagram of a smart hierarchical storage system according to a preferred embodiment of the present invention.
  • the system includes a metadata server cluster, a file access access server cluster, a data storage server cluster, and a smart hierarchical storage engine.
  • the solution of the preferred embodiment of the present invention is implemented by the cooperation of the intelligent hierarchical storage engine and other service clusters.
  • FIG. 13 is a flowchart of a smart tiered storage method according to a preferred embodiment of the present invention.
  • the smart adjustment of high-level storage resource pool expansion, intelligent adjustment of high-level storage resource pool shrinkage, and system hotspot range expansion are described below with reference to FIG. Hardware adjustment suggestions / adaptive adjustments are explained.
  • the smart adjustment for the expansion of the high-level storage resource pool includes the following steps:
  • SSD Solid State Drives
  • the storage resource pool information monitoring and reporting module deployed on the storage server reports the resource pool information of the SSD device that is added to the storage device to the intelligent tiered storage engine, which includes new device information of different hierarchical storage resource pools, including high-level Current storage resource pool current usage capacity, new total capacity, etc.
  • the intelligent tiered storage engine collects the data fragmentation access information reported by the real-time data fragmentation collection and reporting module on the metadata server cluster in the new data fragmentation access statistical period, where the data fragmentation resource is included. The location of the pool, the number of times visited, etc.;
  • the intelligent hierarchical storage engine collects and records the detailed migration information based on the data fragmentation, including the migration direction, reported by the data fragmentation migration and reporting module on the data storage server cluster in the new data fragmentation access statistical period;
  • All data fragmentation information lists LT including current location information and access heat information, which are bidirectionally migrated in all previous consecutive 2 statistical periods recorded according to the method of step (5);
  • the intelligent tiered storage engine collects and counts all the data fragmentation heat information in the system during the new data fragmentation access statistical period, and compares the current migration heat threshold to form a new list of pending migration data fragments LA. And a new migration data fragmentation list LD to be dropped;
  • the high-level resource pool after the expansion is fully utilized, especially for data fragmentation that is frequently bidirectionally migrated before unexpanded, and the jitter migration is effectively suppressed, and the internal migration consumption of data between different hierarchical resource pools is reduced.
  • the performance gain value is increased.
  • the number of hot data fragments included in the high-level storage resource pool is automatically expanded to be more and wider. The system has greatly improved the overall access performance of the upper application.
  • Smart adjustments for high-level storage resource pool shrinkage include the following steps:
  • the storage resource pool information monitoring and reporting module deployed on the storage server reports the new resource pool information of the SSD device abnormality or the SSD device to be extracted to the intelligent hierarchical storage engine, including new devices of different hierarchical storage resource pools.
  • the composition information includes the current usage capacity of the high-level storage resource pool, the new total capacity, and the like.
  • OMM operation and maintenance management
  • the intelligent hierarchical storage engine collects and records the detailed migration information based on the data fragmentation reported by the data fragment migration and reporting module on the data storage server cluster, including the migration direction, in the new data fragmentation access statistical period;
  • the number of hotspot contents of the system does not change when the SSD disk is abnormal (or the SSD disk is removed), but the high-level storage resource pool is caused by the abnormality of the SSD disk (or the SSD disk is pulled out).
  • the capacity becomes smaller, causing the high-level storage resource pool space usage ratio r to rise in each cycle after the change occurs;
  • r1 is less than or equal to the high-level storage resource pool full threshold, the system can do nothing, indicating that the current SSD disk is abnormal (or the SSD disk is unplugged) and does not access the hot content currently stored in the upper-layer storage resource pool. Influencing (declining migration), that is, the performance gain value of the current hot content remains unchanged;
  • r1 is greater than the current threshold of the current high-level resource pool of the system, in order to ensure the normal operation of the high-level resource pool, the system must access the heat record according to the data fragmentation in the new at least two statistical periods, and the high-level resource pool will be The content of the data fragment that occupies the space that exceeds the full threshold of the resource pool is forcibly dropped and migrated to the low-level storage resource pool, resulting in a decrease in system performance improvement revenue;
  • step (10) The content of the bidirectional migration jitter in step (9) will be accurately identified by the data fragmentation access thermal statistics recording module and the data fragmentation migration statistical recording module in the intelligent hierarchical storage engine;
  • Hb may not exist. At this time, it indicates that the data fragmentation capacity that needs to be upgraded in each cycle of the system exceeds the maximum capacity r ⁇ C of the entire upper-level resource pool.
  • the threshold of the system data fragmentation migration is adjusted to: Ha*LA/(r ⁇ C), intelligently completes the thermal threshold adjustment, and dynamically eliminates/reduces the system bidirectional migration jitter caused by the reduction of the high-level resource pool capacity.
  • Intelligent hardware tuning recommendations/adaptive adjustments for system hotspot extensions include the following steps:
  • the hotspot range in the system expands, including the heat statistics period in the time point and subsequent consecutive heat statistics periods to find that the hot content expands, and the high-level storage resource pool is limited.
  • the intelligent tiered storage engine is determined to be frequently swapped out in two or more consecutive periods according to the statistics and records of the bidirectional migration of data fragments in two consecutive periods or more, and the performance improvement gains compared to the decline before the hotspot is expanded.
  • the data fragment set that is swapped in, the required capacity of the slice set is set according to the current storage redundancy mode of the distributed storage system, and the full threshold value is used to calculate the minimum capacity and the optimal capacity that the upper-level storage resource pool needs to expand. ;
  • the intelligent grading engine sets a timeout timer while the feedback of the high-level storage resource pool is extended due to the expansion of the hotspot range. Before the timeout timer expires, the system administrator expands according to the recommendation. The system automatically performs the following steps: "Intelligent adjustment of high-level storage resource pool expansion".
  • Hb may not exist. At this time, it indicates that the data fragmentation capacity that needs to be upgraded in each cycle of the system exceeds the maximum capacity r ⁇ C of the entire upper-level resource pool.
  • the system data fragmentation migration threshold is adjusted to: Ha * LA / (r ⁇ C);
  • modules or steps of the present invention described above can be implemented by a general-purpose computing device that can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they may be implemented by program code executable by the computing device such that they may be stored in the storage device by the computing device and, in some cases, may be different from the order herein.
  • the steps shown or described are performed, or they are separately fabricated into individual integrated circuit modules, or a plurality of modules or steps thereof are fabricated as a single integrated circuit module.
  • the invention is not limited to any specific combination of hardware and software.
  • the foregoing technical solution provided by the embodiment of the present invention may be applied to the process of establishing a capacity change, and the number of two-way migration between the first storage resource pool and the second storage resource pool on the data storage server cluster in the preset time period is more than
  • the third data fragment of the migration number is set. According to the data volume of the third data fragment, the manner of recommending the capacity change of the first storage resource pool is reported, and the service performance degradation of the migration policy caused by manual static setting is solved. The problem is to improve the access performance of the business.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Debugging And Monitoring (AREA)

Abstract

L'invention concerne un procédé et un dispositif de suggestion de changement de capacité. Le procédé comprend : le comptage de troisièmes fragments de données, la fréquence de migrations bidirectionnelles entre un premier et un second groupe de ressources de mémorisation sur une grappe de serveurs de mémorisation de données dépassant un nombre de migrations prédéfini au cours d'une période prédéfinie; et, en fonction de la quantité de données des troisièmes fragments de données, la transmission d'une suggestion de changement de capacité pour le premier groupe de ressources de mémorisation. La présente invention résout le problème de performances d'accès réduites à un service en raison d'une politique de migration fixée manuellement et de manière statique, et améliore les performances d'accès à un service.
PCT/CN2016/104851 2015-09-06 2016-11-07 Procédé et dispositif de suggestion de changement de capacité WO2017036428A2 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510698063.3 2015-09-06
CN201510698063.3A CN106502578B (zh) 2015-09-06 2015-09-06 容量变更建议方法及装置

Publications (2)

Publication Number Publication Date
WO2017036428A2 true WO2017036428A2 (fr) 2017-03-09
WO2017036428A3 WO2017036428A3 (fr) 2017-04-13

Family

ID=58186719

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/104851 WO2017036428A2 (fr) 2015-09-06 2016-11-07 Procédé et dispositif de suggestion de changement de capacité

Country Status (2)

Country Link
CN (1) CN106502578B (fr)
WO (1) WO2017036428A2 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110008199A (zh) * 2019-03-25 2019-07-12 华南理工大学 一种基于访问热度的数据迁移部署方法
CN110825908A (zh) * 2019-11-04 2020-02-21 安超云软件有限公司 一种对象的迁移方法、装置、电子设备及存储介质
CN111124692A (zh) * 2020-01-02 2020-05-08 神州数码融信软件有限公司 一种业务请求处理***

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106502576B (zh) * 2015-09-06 2020-06-23 中兴通讯股份有限公司 迁移策略调整方法及装置
CN112948398B (zh) * 2021-04-29 2023-02-24 电子科技大学 一种面向冷热数据的分级存储***及方法
CN113590586B (zh) * 2021-07-29 2022-03-22 东方微银科技股份有限公司 分布式图数据库***节点间迁移分片数据的方法、装置
CN114936003B (zh) * 2022-05-06 2023-03-21 北京新科安云信息技术有限公司 资源池的数据分层迁移方法、装置、设备及可读存储介质
CN115373594A (zh) * 2022-07-12 2022-11-22 浙江大学 一种双向动态切换的存储***及动态切换管理方法

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102143206A (zh) * 2010-12-17 2011-08-03 浪潮(北京)电子信息产业有限公司 集群存储***中存储池的调整方法、装置及***
CN102043732A (zh) * 2010-12-30 2011-05-04 成都市华为赛门铁克科技有限公司 一种缓存分配方法及装置
CN103365781B (zh) * 2012-03-29 2016-05-04 国际商业机器公司 用于动态地重新配置存储***的方法和设备
CN103150263B (zh) * 2012-12-13 2016-01-20 深圳先进技术研究院 分级存储方法
CN103078933B (zh) * 2012-12-29 2015-12-02 深圳先进技术研究院 一种确定数据迁移时机的方法和装置
CN104657215A (zh) * 2013-11-19 2015-05-27 南京鼎盟科技有限公司 云计算中虚拟化节能***
CN104660578B (zh) * 2014-04-22 2017-12-19 董唯元 一种实现数据安全存储及数据访问控制的***及其方法
CN104536994A (zh) * 2014-12-11 2015-04-22 北京京东尚科信息技术有限公司 通用的数据迁移方法和装置

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110008199A (zh) * 2019-03-25 2019-07-12 华南理工大学 一种基于访问热度的数据迁移部署方法
CN110825908A (zh) * 2019-11-04 2020-02-21 安超云软件有限公司 一种对象的迁移方法、装置、电子设备及存储介质
CN111124692A (zh) * 2020-01-02 2020-05-08 神州数码融信软件有限公司 一种业务请求处理***
CN111124692B (zh) * 2020-01-02 2023-05-12 神州数码融信软件有限公司 一种业务请求处理***

Also Published As

Publication number Publication date
CN106502578B (zh) 2019-06-11
CN106502578A (zh) 2017-03-15
WO2017036428A3 (fr) 2017-04-13

Similar Documents

Publication Publication Date Title
WO2016165441A1 (fr) Procédé d'ajustement de politique de migration, dispositif et procédé de suggestion de changement de capacité
WO2017036428A2 (fr) Procédé et dispositif de suggestion de changement de capacité
US9355112B1 (en) Optimizing compression based on data activity
CN100518088C (zh) 一种流媒体数据内容管理方法
CN104978362B (zh) 分布式文件***的数据迁移方法、装置及元数据服务器
US20230004436A1 (en) Container scheduling method and apparatus, and non-volatile computer-readable storage medium
US9727487B2 (en) Cache management method and apparatus for non-volatile storage device
CN103576835B (zh) 一种休眠磁盘的数据操作方法和装置
US20120221730A1 (en) Resource control system and resource control method
US20150199281A1 (en) Apparatus and method for meeting performance metrics for users in file systems
CN103631666B (zh) 数据冗余容错的适配管理设备、服务设备、***及方法
CN103078933B (zh) 一种确定数据迁移时机的方法和装置
CN109085995B (zh) 数据动态分片的存储方法、装置和***
US11157313B2 (en) Method, apparatus and computer storage medium for controlling a storage system
CN104092620A (zh) 一种实现网络带宽调整的方法及装置
CN111414070A (zh) 一种机箱功耗管理方法、***及电子设备和存储介质
WO2019011262A1 (fr) Procédé et appareil d'attribution de ressources
WO2016082176A1 (fr) Procédé et dispositif associé de prédiction de seuil de données
CN111143279A (zh) 一种数据迁移方法、装置、设备及可读存储介质
WO2022017290A1 (fr) Procédé et appareil de commande de vitesse de réseau d'un terminal, terminal et support de stockage
CN111506425B (zh) 服务质量数据处理方法和装置
EP3076295A1 (fr) Procédé et dispositif d'ajustement de l'équilibre des ressources d'un système
CN104239230B (zh) 一种数据块迁移方法及装置
CN103530240B (zh) 数据块缓存方法和装置
US20200387628A1 (en) Method and device for improving data storage security

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16840857

Country of ref document: EP

Kind code of ref document: A2

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16840857

Country of ref document: EP

Kind code of ref document: A2