CN114546259A - Multi-level storage system based on income - Google Patents

Multi-level storage system based on income Download PDF

Info

Publication number
CN114546259A
CN114546259A CN202111632251.8A CN202111632251A CN114546259A CN 114546259 A CN114546259 A CN 114546259A CN 202111632251 A CN202111632251 A CN 202111632251A CN 114546259 A CN114546259 A CN 114546259A
Authority
CN
China
Prior art keywords
storage
data
service
cost
metadata
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111632251.8A
Other languages
Chinese (zh)
Inventor
尹春光
应宇晖
吴晓兵
张璐波
闻一波
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tianyi Electronic Commerce Co Ltd
Original Assignee
Tianyi Electronic Commerce Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tianyi Electronic Commerce Co Ltd filed Critical Tianyi Electronic Commerce Co Ltd
Priority to CN202111632251.8A priority Critical patent/CN114546259A/en
Publication of CN114546259A publication Critical patent/CN114546259A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0604—Improving or facilitating administration, e.g. storage management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/0647—Migration mechanisms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
    • G06Q10/06312—Adjustment or analysis of established resource schedule, e.g. resource or task levelling, or dynamic rescheduling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
    • G06Q10/06313—Resource planning in a project environment

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Theoretical Computer Science (AREA)
  • Human Resources & Organizations (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Strategic Management (AREA)
  • Human Computer Interaction (AREA)
  • Entrepreneurship & Innovation (AREA)
  • General Engineering & Computer Science (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Tourism & Hospitality (AREA)
  • Development Economics (AREA)
  • General Business, Economics & Management (AREA)
  • Educational Administration (AREA)
  • Game Theory and Decision Science (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Financial Or Insurance-Related Operations Such As Payment And Settlement (AREA)

Abstract

The invention discloses a multi-level storage system based on income, which comprises the following components and an implementation method: the data service profit controller is mainly responsible for collecting the profit factors fed back from different organizations and storing the profit factors into the metadata service; and the metadata service is mainly responsible for metadata storage of the cluster data blocks, data service profit calculation and dynamic scheduling of storage and migration of the data blocks according to profit. The invention designs the storage system from the new definition of the data service income from the new angle, and achieves the purpose of reducing the enterprise data storage cost by dynamically controlling the income created by the data service and the storage cost in a grading manner.

Description

Multi-level storage system based on income
Technical Field
The invention relates to the field of payment and finance, in particular to a multi-level storage system based on income.
Background
The big data age has come, the whole data volume stored by enterprises is continuously increased along with the increase of time and the increase of users, the data volume reaches the PB level, and the data storage cost of the enterprises is continuously increased. All enterprises are exploring and mining data values, so that cost is reduced, and values brought by data are increased. The invention realizes the effective storage of data from a new angle. Different data have different values, different storages stored in a unit have high and low prices, the system mainly introduces a revenue model of the storage system and a multi-level dynamic storage system based on the model to realize revenue maximization, and automatically and dynamically adjusts the storages to save the storage cost of enterprise data.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provide a profit-based multi-level storage system, different storage media stored in a unit have different costs, the system mainly introduces a profit model of the storage system and a multi-level dynamic storage system based on the model, realizes the profit maximization by dynamically controlling the profit and the storage cost of data service, and automatically and dynamically adjusts a storage to realize the saving of the data storage cost of enterprises.
The invention provides the following technical scheme:
the invention provides a multi-level storage system based on income, which comprises the following components and an implementation method:
data service income controller
The data service revenue controller is mainly responsible for collecting revenue factors fed back by different organizations and storing the revenue factors into the metadata service;
two, metadata service
The metadata service is mainly responsible for storing metadata of the cluster data block, calculating the profit of the data service and dynamically performing storage and migration of the scheduling data block according to the profit;
and (3) metadata storage:
data format of metadata: (block _ id, node _ id, value, size);
the metadata information synchronization flow is shown in fig. 2;
calculating the data service income, wherein the calculation formula is that the data service income is the system cost which is the income
Data service revenue:
(1) the data block contains the service attribute and the weight through a group of application data labels tag [ ];
(2) time (time of creation date to date) of the data block;
(3) an initial value β of the data block;
(4) the base attribute of the data block, base _ tag [ ];
(5) a correction coefficient alpha;
the data service income, namely dataV + dataV1+ dataV2+. + dataVn, is obtained by a value calculation function dataV [ ] + base _ tag [ ] + beta) × (time), and is stored in the metadata service;
the system cost is as follows:
(1) the service software and hardware fixed cost of the system comprises storage cost and other software and hardware cost;
(2) cost of Cost;
(3) storing the time t;
(4) a unit storage cost C;
(5) a storage size b;
(6) hardware service period s
Storing the cost b t C/s;
storage node
The storage nodes are data storage nodes in the system, the internal design of the storage nodes is that storage equipment containing different unit costs is adopted, the storage nodes contain 0-N levels, each level of storage has own unit storage cost C (element/GB), data blocks can be correspondingly stored to the corresponding storage levels according to values, and the internal structure is shown in figure 3;
when a data node and a metadata service synchronize metadata of a data block regularly, a data service profit value is obtained, comparison is carried out inside the node according to data service profit and cost of storage at all levels, a storage level interval of the data block is determined, direct migration is carried out to a corresponding storage level, and subsequent data reading and writing can be carried out from the latest data storage of the data block;
example of internal migration flow of data storage node:
(1) the storage level of the blue data block as shown in FIG. 4 is reduced from the storage value of level 1 and then stored to level 3;
(2) the storage level of the red data block as shown in FIG. 4 is increased from the storage value of 2 levels to 1 level;
the storage level of the data is influenced based on the dynamic data service income, and the income grading storage system is realized.
Compared with the prior art, the invention has the following beneficial effects:
the distributed storage represented by HDFS is suitable for a scene of writing in and reading out for many times, but the distributed storage has no different storage for data and no different treatment for core data and edge data, and brings serious burden to the storage cost of enterprises. In fact, the exchange of data in different organizations is realized by creating income, different costs of storage media are different, and the invention designs a storage system from the new defined data service income from a new perspective, and achieves the purpose of reducing the enterprise data storage cost by dynamically and dynamically controlling the income created by the data service and the storage cost in a grading manner.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention. In the drawings:
FIG. 1 is an overall architecture diagram of the present invention;
FIG. 2 is a metadata information synchronization flow diagram of the present invention;
FIG. 3 is a diagram of the internal structure of the multilevel memory of the present invention;
FIG. 4 is a flow chart of the data storage node internal migration of the present invention.
Detailed Description
The preferred embodiments of the present invention will be described in conjunction with the accompanying drawings, and it will be understood that they are described herein for the purpose of illustration and explanation and not limitation. Wherein like reference numerals refer to like parts throughout.
Example 1
1-4, the present invention provides a revenue-based multi-level storage system, comprising the following components and methods of implementation:
data service income controller
The data service revenue controller is mainly responsible for collecting revenue factors fed back by different organizations and storing the revenue factors into the metadata service;
two, metadata service
The metadata service is mainly responsible for storing metadata of the cluster data block, calculating the profit of the data service and dynamically performing storage and migration of the scheduling data block according to the profit;
and (3) metadata storage:
data format of metadata: (block _ id, node _ id, value, size);
the metadata information synchronization flow is shown in fig. 2;
calculating the data service income, wherein the calculation formula is that the data service income is the system cost which is the income
Data service revenue:
(1) the data block contains the service attribute and the weight through a group of application data labels tag [ ];
(2) time (time of creation date to date) of the data block;
(3) an initial value β of the data block;
(4) the base attribute of the data block, base _ tag [ ];
(5) a correction coefficient alpha;
the data service income, namely dataV + dataV1+ dataV2+. + dataVn, is obtained by a value calculation function dataV [ ] + base _ tag [ ] + beta) × (time), and is stored in the metadata service;
the system cost is as follows:
(1) the service software and hardware fixed cost of the system comprises storage cost and other software and hardware cost;
(2) cost of Cost;
(3) storing the time t;
(4) a unit storage cost C;
(5) a storage size b;
(6) hardware service period s
Storing the cost b t C/s;
storage node
The storage nodes are data storage nodes in the system, the internal design of the storage nodes is that storage equipment containing different unit costs is adopted, the storage nodes contain 0-N levels, each level of storage has own unit storage cost C (element/GB), data blocks can be correspondingly stored to the corresponding storage levels according to values, and the internal structure is shown in figure 3;
when a data node and a metadata service synchronize metadata of a data block regularly, a data service profit value is obtained, comparison is carried out inside the node according to data service profit and cost of storage at all levels, a storage level interval of the data block is determined, direct migration is carried out to a corresponding storage level, and subsequent data reading and writing can be carried out from the latest data storage of the data block;
example of internal migration flow of data storage node:
(1) the storage level of the blue data block as shown in FIG. 4 is reduced from the storage value of level 1 and then stored to level 3;
(2) the storage level of the red data block as shown in FIG. 4 is increased from the storage value of 2 levels to 1 level;
the storage level of the data is influenced based on the dynamic data service income, and the income grading storage system is realized.
As shown in FIG. 1, the present patent invention is a revenue-based multi-level storage system.
As shown in fig. 1, the assembly comprises: data service revenue controller, metadata service, storage device
1. Starting all components in the storage system;
2. receiving the requirement of a user for feeding back data service;
3. providing a data service system for a business party to use;
and the storage system dynamically performs data service income statistics and storage level adjustment in real time.
Finally, it should be noted that: although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that changes may be made in the embodiments and/or equivalents thereof without departing from the spirit and scope of the invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (1)

1. A multi-level revenue-based storage system, comprising the following components and methods of implementation:
data service income controller
The data service revenue controller is mainly responsible for collecting revenue factors fed back by different organizations and storing the revenue factors into the metadata service;
two, metadata service
The metadata service is mainly responsible for storing metadata of the cluster data block, calculating the profit of the data service and dynamically performing storage and migration of the scheduling data block according to the profit;
and (3) metadata storage:
data format of metadata: (block _ id, node _ id, value, size);
calculating the data service income, wherein the calculation formula is that the data service income is the system cost which is the income
Data service revenue:
(1) the data block contains the service attribute and the weight through a group of application data labels tag [ ];
(2) time (time of creation date to date) of the data block;
(3) an initial value β of the data block;
(4) the base attribute of the data block, base _ tag [ ];
(5) a correction coefficient alpha;
the data service income, namely dataV + dataV1+ dataV2+. + dataVn, is obtained by a value calculation function dataV [ ] + base _ tag [ ] + beta) × (time), and is stored in the metadata service;
the system cost is as follows:
(1) the service software and hardware fixed cost of the system comprises storage cost and other software and hardware cost;
(2) cost of Cost;
(3) storing the time t;
(4) a unit storage cost C;
(5) a storage size b;
(6) hardware service period s
Storing the cost b t C/s;
storage node
The storage nodes are data storage nodes in the system, the internal design of the storage nodes is that storage equipment containing different unit costs is adopted, the storage nodes contain 0-N levels, the storage of each level has own unit storage cost C (element/GB), and data blocks can be correspondingly stored to the corresponding storage levels according to values;
when the data node and the metadata service synchronize the metadata of the data block regularly, the data service profit value is obtained, comparison is carried out inside the node according to the data service profit and the cost of each level of storage, the storage level interval of the data block is determined, direct migration is carried out to the corresponding storage level, and subsequent data reading and writing can be carried out from the latest data storage of the data block.
CN202111632251.8A 2021-12-28 2021-12-28 Multi-level storage system based on income Pending CN114546259A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111632251.8A CN114546259A (en) 2021-12-28 2021-12-28 Multi-level storage system based on income

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111632251.8A CN114546259A (en) 2021-12-28 2021-12-28 Multi-level storage system based on income

Publications (1)

Publication Number Publication Date
CN114546259A true CN114546259A (en) 2022-05-27

Family

ID=81670501

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111632251.8A Pending CN114546259A (en) 2021-12-28 2021-12-28 Multi-level storage system based on income

Country Status (1)

Country Link
CN (1) CN114546259A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115098023A (en) * 2022-06-22 2022-09-23 浙江威固信息技术有限责任公司 Array memory and memory unit control method

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115098023A (en) * 2022-06-22 2022-09-23 浙江威固信息技术有限责任公司 Array memory and memory unit control method
CN115098023B (en) * 2022-06-22 2023-08-15 浙江威固信息技术有限责任公司 Array memory and memory unit control method

Similar Documents

Publication Publication Date Title
US9697571B2 (en) Real-time file system charge-back accounting per management object during a report cycle
CN104850598B (en) A kind of real-time data base back-up restoring method
CN101436207B (en) Data restoring and synchronizing method based on log snapshot
CN100547583C (en) Database automatically and the method that dynamically provides
US7613747B1 (en) Tiered database storage and replication
CN100410944C (en) Method, system and computer program product for backing up and recovering a database
US20080306990A1 (en) System for performing a redistribute transparently in a multi-node system
CN106708427A (en) Storage method suitable for key value pair data
CN101777016A (en) Snapshot storage and data recovery method of continuous data protection system
CN107193985A (en) A kind of slide fastener table design method of record data change histories
CN114546259A (en) Multi-level storage system based on income
CN106445409A (en) Distributed block storage data writing method and device
CN116737064B (en) Data management method and system for solid state disk
US20190227928A1 (en) Cost-based garbage collection scheduling in a distributed storage environment
CN101719131B (en) Method, device and system for processing report data
CN102122284B (en) Compound document storage and read-write method and compound document storage and read-write device
CN102012946A (en) High-efficiency safety monitoring video/image data storage method
CN111163149B (en) Intelligent contract platform method based on block chain
CN107506466A (en) A kind of small documents storage method and system
CN102902694A (en) Picture checking method and device
CN101303657A (en) Method of optimization of multiprocessor real-time task execution power consumption
CN114253917A (en) Distributed self-adaptive storage method and system based on file access characteristics
CN102222280A (en) Information interaction processing system and method
CN104142943A (en) Database expansion method and database
CN103778231A (en) Method and system for managing operation record information, of databank

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication