CN115988008A - High-density storage method and system for cloud storage system - Google Patents
High-density storage method and system for cloud storage system Download PDFInfo
- Publication number
- CN115988008A CN115988008A CN202211705927.6A CN202211705927A CN115988008A CN 115988008 A CN115988008 A CN 115988008A CN 202211705927 A CN202211705927 A CN 202211705927A CN 115988008 A CN115988008 A CN 115988008A
- Authority
- CN
- China
- Prior art keywords
- storage
- data
- module
- node
- unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 26
- 238000012544 monitoring process Methods 0.000 claims abstract description 29
- 230000005540 biological transmission Effects 0.000 claims abstract description 16
- 230000006835 compression Effects 0.000 claims abstract description 11
- 238000007906 compression Methods 0.000 claims abstract description 11
- 238000001514 detection method Methods 0.000 claims abstract description 9
- 238000004891 communication Methods 0.000 claims description 3
- 238000013500 data storage Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of cloud storage, and discloses a high-density storage method and a high-density storage system for a cloud storage system, wherein the high-density storage method and the high-density storage system comprise a client, a storage node, a storage unit, a backup unit and a monitoring control node; the Ceph OSD storage node is connected with the storage unit and the backup unit through the SAS line, the monitoring control node firstly detects data uploaded by a user through the detection module in the data transmission process to ensure the safety of the data, and then compresses the data through the compression module and records the data through the recording module; the user can mark important data, the monitoring control node stores the marked important data into the backup unit and the storage unit through the transmission module and the storage node in a backup mode, and transmits the compressed data into the storage module in the storage unit, so that the storage performance of the data is better, and the data is safer and more reliable to store.
Description
Technical Field
The invention belongs to the technical field of cloud storage, and particularly relates to a high-density storage method and system for a cloud storage system.
Background
Cloud storage is a mode of online storage (english: cloud storage), i.e., data is stored on multiple virtual servers, usually hosted by third parties, rather than on dedicated servers. A hosting company operates a large-scale data center, and people who need data storage hosting can meet the requirement of data storage by purchasing or leasing storage space from the people. The data center operator prepares the storage virtualized resources at the back end according to the needs of the customer, and provides the resources in a storage resource pool (storage pool), so that the customer can use the storage resource pool to store the files or objects by himself. In fact, these resources may be distributed across numerous server hosts, cloud storage, also known as cloud backup, is a computing service model that lets you transfer, store, and access files and data remotely, over the internet, all provided by domestic or overseas cloud host vendors. Cloud providers operate and manage data storage services, shifting responsibility for data processing away from customers. The responsibilities further include hosting, protecting, maintaining and managing the server and related infrastructure, and in order to improve the storage performance and storage installation of the cloud storage, a high-density storage method and system for the cloud storage system are provided.
Disclosure of Invention
The invention aims to provide a high-density storage method and a high-density storage system for a cloud storage system, so that the data storage performance is better and the storage is safer and more reliable.
The technical scheme adopted by the invention is as follows: a high-density storage method and a system for a cloud storage system specifically comprise the following steps:
s1: firstly, initializing, connecting a storage node with a storage unit and a backup unit, and then recording and numbering the storage unit and the backup unit;
s2: the storage node is connected with the monitoring control node, and a user uploads and stores data to the storage unit through the client;
s3: in the data transmission process, the monitoring control node firstly detects data to ensure the safety of the data and then compresses and records the data;
s4: the user can mark the important data, and the monitoring control node stores the marked important data into the backup unit through the storage node backup;
s5: the storage unit and the backup unit are in network communication with the monitoring control node through the storage node;
s6: when a user downloads the cloud file through the client, the monitoring control node detects the data in the storage unit through the storage node, and if the marked important data in the storage unit is found to be invalid, S7 is executed;
s7: and extracting the failure data from the storage unit, comparing, copying and intercepting from the backup unit, decoding the compressed data, and sending the decoded data to the user client to finish the data access process.
Optionally, the storage node is a Ceph OSD storage node.
Optionally, the storage nodes may be connected to a plurality of storage units and a plurality of backup units, and the number of the storage nodes is the same as that of the backup units.
Optionally, each storage unit has a backup unit with the same number.
Optionally, the storage unit and the backup unit both include a storage module and a transmission module, the storage node includes a network module and an encoding module, the monitoring control node includes a compression module, a decoding module, a recording module and a detection module, and the monitoring control node also includes a network module and a transmission module.
Optionally, the storage module is configured to store data, and the transmission module is configured to transmit data.
Optionally, the encoding module is configured to number the storage unit and the backup unit, and the network module is used to connect to a connection port of a network cable.
Optionally, the compression module is configured to compress data, the detection module is configured to detect the data to ensure security of the data, the recording module is configured to record marked important data, and the decoding module is configured to decode and recover the compressed data.
Optionally, the storage node is connected with the storage unit and the backup unit through an SAS line, the storage node is connected with the monitoring control node through a network module and a network line, and the client is connected with the monitoring control node through a switch.
The invention has the technical effects that:
the scheme adopts the Ceph OSD storage node to be connected with the storage unit and the backup unit through the SAS line, the monitoring control node firstly detects data uploaded by a user through the detection module in the data transmission process to ensure the safety of the data, and then compresses the data through the compression module and records the data through the recording module; the user can mark important data, the monitoring control node backs up and stores the marked important data into the backup unit and the storage unit through the transmission module and the storage node, and transmits the compressed data into the storage module in the storage unit, so that the storage performance of the data is better, and the data is stored more safely and reliably;
the Ceph OSD storage node is adopted, storage work is completed by an OSD Daemon process, a Cluster generally comprises the OSD nodes, for reading and writing object data, a client firstly acquires Cluster Map information from a Monitor node, and then directly interacts with OSD for I/O operation, the reading and writing process is accelerated.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart of the method of the present invention;
FIG. 2 is a system diagram of the present invention.
In the drawings, the components represented by the respective reference numerals are listed below:
1. a storage module; 2. a transmission module; 3. a network module; 4. an encoding module; 5. a recording module; 6. a detection module; 7. a compression module; 8. and a decoding module.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1-2, the present invention provides a high density storage method and system for a cloud storage system, which specifically includes the following steps:
firstly, initializing, connecting a Ceph OSD storage node with a storage unit and a backup unit through SAS lines, recording and numbering the storage unit and the backup unit through a coding module 4, wherein each storage unit is provided with a backup unit with the same number; the storage node is connected with the monitoring control node, a user uploads and stores data to the storage unit through a client, the client is connected with the switch through a network cable, and the switch is connected with the monitoring control node and the Ceph OSD storage node through the network cable; in the data transmission process, the monitoring control node firstly detects data uploaded by a user through a detection module 6 to ensure the safety of the data, then compresses the data through a compression module 7 and records the data through a recording module 5; the user can mark important data, the monitoring control node stores the marked important data into the backup unit and the storage module 1 through the transmission module 2 and the storage node, and transmits the compressed data into the storage module 1 in the storage unit; the storage unit and the backup unit are in network communication with the monitoring control node through the transmission module 2 and the network module 3 in the storage node; when a user downloads a cloud file through a client, the monitoring control node detects data inside the storage module 1 in the storage unit through the detection module 6, if the fact that important data marked in the storage unit is invalid or damaged is found, corresponding data recorded through the recording module 5 is executed to extract invalid data from the storage module 1 in the storage unit, the invalid data is compared and copied and intercepted from the storage module 1 in the backup unit, the data and normal data are decoded through the decoding module 8 and then sent to the switch, the data and the switch are sent to the user client through the switch, and the data access process is completed, so that the storage performance of the data is better, and the data is stored more safely and reliably;
ceph OSD storage node: the Ceph OSD is the most important component of the Ceph storage Cluster, the Ceph OSD stores data on a physical disk of each node in the Cluster in an object form, the storage work is completed by an OSD Daemon process, the Cluster generally comprises OSD nodes, for the reading and writing of the object data, a client firstly obtains Cluster Map information from a Monitor node, and a subsequent client directly interacts with the OSD in an I/O operation mode, so that the reading and writing process is accelerated, compared with other distributed storage systems, the efficiency is higher, the core function of the Ceph comprises high reliability, automatic balance, automatic recovery and consistency, and the Ceph OSD storage node is adopted to ensure that the data is more reliable and stable when being transmitted and stored;
a network switch: the whole cluster is connected by adopting a universal switch to provide cloud storage service to the outside;
the compression module 7: by compressing the file data, the compression ratio of the general character files is higher and can reach about 50%, and the compression ratio of the video, audio and image files is about 80%, so that the occupied space of the files can be reduced, and the storage efficiency can be improved.
The preferred embodiments of the invention disclosed above are intended to be illustrative only. The preferred embodiments are not intended to be exhaustive or to limit the invention to the precise embodiments disclosed. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and the practical application, to thereby enable others skilled in the art to best utilize the invention. The invention is limited only by the claims and their full scope and equivalents.
Claims (9)
1. A high-density storage method for a cloud storage system is applied to cloud storage and is characterized by comprising the following steps:
s1: firstly, initializing, connecting a storage node with a storage unit and a backup unit, and then recording and numbering the storage unit and the backup unit;
s2: the storage node is connected with the monitoring control node, and a user uploads and stores data to the storage unit through the client;
s3: in the data transmission process, the monitoring control node firstly detects data to ensure the safety of the data and then compresses and records the data;
s4: the user can mark the important data, and the monitoring control node stores the marked important data into the backup unit through the backup of the storage node;
s5: the storage unit and the backup unit are in network communication with the monitoring control node through the storage node;
s6: when a user downloads a cloud file through a client, the monitoring control node detects data in the storage unit through the storage node, and if important data marked in the storage unit are found to be invalid, S7 is executed;
s7: and extracting the failure data from the storage unit, comparing, copying and intercepting from the backup unit, decoding the compressed data, and sending the decoded data to the user client to finish the data access process.
2. The high-density storage method for the cloud storage system according to claim 1, wherein: the storage node adopts a Ceph OSD storage node.
3. The high-density storage method for the cloud storage system according to claim 1, wherein: the storage nodes can be connected with a plurality of storage units and a plurality of backup units, and the number of the storage nodes is the same as that of the backup units.
4. The high-density storage method for the cloud storage system according to claim 1, wherein: each storage unit has a backup unit with the same number.
5. An operating system for a high density storage method of a cloud storage system according to any one of claims 1 to 4, wherein: the storage unit and the backup unit respectively comprise a storage module (1) and a transmission module (2), the storage node comprises a network module (3) and an encoding module (4), the monitoring control node comprises a compression module (7), a decoding module (8), a recording module (5) and a detection module (6), and the monitoring control node also comprises the network module (3) and the transmission module (2).
6. The operating system of claim 5, wherein the operating system is configured to perform the high-density storage method according to one of the following: the storage module (1) is used for storing data, and the transmission module (2) is used for transmitting data.
7. The operating system of claim 5, wherein the operating system is configured to perform the high-density storage method according to one of the following: the coding module (4) is used for numbering the storage units and the backup units, and the network module (3) is used for connecting a connection port of a network cable.
8. The operating system of claim 5, wherein the operating system is configured to perform the high-density storage method according to one of the following: the compression module (7) is used for compressing data, the detection module (6) is used for detecting the data to ensure the safety of the data, the recording module (5) is used for recording marked important data, and the decoding module (8) is used for decoding and recovering the compressed data.
9. The operating system of claim 5, wherein the operating system is configured to perform the high-density storage method according to one of the following: the storage node is connected with the storage unit and the backup unit through SAS lines, the storage node is connected with the monitoring control node through a network module (3) and a network line, and the client is connected with the monitoring control node through a switch.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211705927.6A CN115988008A (en) | 2022-12-29 | 2022-12-29 | High-density storage method and system for cloud storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211705927.6A CN115988008A (en) | 2022-12-29 | 2022-12-29 | High-density storage method and system for cloud storage system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115988008A true CN115988008A (en) | 2023-04-18 |
Family
ID=85959284
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211705927.6A Pending CN115988008A (en) | 2022-12-29 | 2022-12-29 | High-density storage method and system for cloud storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115988008A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1455276A1 (en) * | 2003-03-06 | 2004-09-08 | Hewlett-Packard Development Company, L.P. | Data processing system and method to restore a backup |
| CN106060173A (en) * | 2016-07-22 | 2016-10-26 | 恒业智能信息技术（深圳）有限公司 | Cluster type photographing data storage system based on cloud storage |
| CN205961187U (en) * | 2016-08-22 | 2017-02-15 | 成都信息工程大学 | Cloud storage system with apart from monitoring devices |
| CN107046575A (en) * | 2017-04-18 | 2017-08-15 | 南京卓盛云信息科技有限公司 | A kind of cloud storage system and its high density storage method |
| CN107332939A (en) * | 2017-09-02 | 2017-11-07 | 刘兴丹 | It is a kind of to pass through the cloud storage method that is locally stored, device |
| CN107704339A (en) * | 2012-12-26 | 2018-02-16 | 华为终端有限公司 | A kind of method, apparatus and user equipment of media file backup |
| CN115220666A (en) * | 2022-09-21 | 2022-10-21 | 南京中岱得存储技术有限公司 | Independent cloud storage device and data circulation method applying same |
-
2022
- 2022-12-29 CN CN202211705927.6A patent/CN115988008A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1455276A1 (en) * | 2003-03-06 | 2004-09-08 | Hewlett-Packard Development Company, L.P. | Data processing system and method to restore a backup |
| CN107704339A (en) * | 2012-12-26 | 2018-02-16 | 华为终端有限公司 | A kind of method, apparatus and user equipment of media file backup |
| CN106060173A (en) * | 2016-07-22 | 2016-10-26 | 恒业智能信息技术（深圳）有限公司 | Cluster type photographing data storage system based on cloud storage |
| CN205961187U (en) * | 2016-08-22 | 2017-02-15 | 成都信息工程大学 | Cloud storage system with apart from monitoring devices |
| CN107046575A (en) * | 2017-04-18 | 2017-08-15 | 南京卓盛云信息科技有限公司 | A kind of cloud storage system and its high density storage method |
| CN107332939A (en) * | 2017-09-02 | 2017-11-07 | 刘兴丹 | It is a kind of to pass through the cloud storage method that is locally stored, device |
| CN115220666A (en) * | 2022-09-21 | 2022-10-21 | 南京中岱得存储技术有限公司 | Independent cloud storage device and data circulation method applying same |
Non-Patent Citations (1)
| Title |
|---|
| 徐继业等: "《气象大数据》", 《上海科学技术出版社》, pages: 60 - 61 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10305988B2 (en) | Adaptive resource utilization with request cancellation | |
| US10481978B2 (en) | Optimal slice encoding strategies within a dispersed storage unit | |
| CN103226502B (en) | A kind of data calamity is for control system and data reconstruction method | |
| US8255653B2 (en) | System and method for adding a storage device to a cluster as a shared resource | |
| US10013166B2 (en) | Virtual tape library system | |
| CN102937967A (en) | Data redundancy realization method and device | |
| CN109412755B (en) | Multimedia data processing method, device and storage medium | |
| CN105550073A (en) | Database backup method and system thereof | |
| CN102014152A (en) | Long-distance duplicating system and method | |
| US10268374B2 (en) | Redundant array of independent discs and dispersed storage network system re-director | |
| CN111522499B (en) | Operation data reading device and reading method thereof | |
| US9817834B1 (en) | Techniques for performing an incremental backup | |
| CN102364472A (en) | Data storage method and system | |
| CN107861843A (en) | A kind of file backup system based on LAN computer | |
| CN106843760A (en) | It is a kind of based on the asynchronous remote copy system deleted and method again | |
| CN106095618A (en) | The method and system of data manipulation | |
| CN115988008A (en) | High-density storage method and system for cloud storage system | |
| US7353354B2 (en) | Application-based commit for local storage subsystems and remote storage subsystems | |
| CN107888430B (en) | Virtual machine management operation synchronization method and device | |
| CN109542519A (en) | A kind of automatic distributed system method for reconstructing | |
| CN100592299C (en) | Management method and storage system of file information in storage system | |
| CN103180831A (en) | Fault handling systems and methods | |
| CN107229535B (en) | Multi-copy storage method, storage device and data reading method for data block | |
| CN101454835B (en) | Image recording/reproducing device | |
| CN114390007B (en) | Data processing method, device, storage medium and processor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |