CN117785800A - Distributed file content management method, device, system and storage medium - Google Patents
Distributed file content management method, device, system and storage medium Download PDFInfo
- Publication number
- CN117785800A CN117785800A CN202410024966.2A CN202410024966A CN117785800A CN 117785800 A CN117785800 A CN 117785800A CN 202410024966 A CN202410024966 A CN 202410024966A CN 117785800 A CN117785800 A CN 117785800A
- Authority
- CN
- China
- Prior art keywords
- content
- target
- manager
- distributed file
- operation instruction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007726 management method Methods 0.000 title claims abstract description 84
- 238000000034 method Methods 0.000 claims abstract description 24
- 230000001360 synchronised effect Effects 0.000 claims description 9
- 230000007246 mechanism Effects 0.000 claims description 8
- 230000006870 function Effects 0.000 abstract description 7
- 238000011161 development Methods 0.000 abstract description 5
- 206010010099 Combined immunodeficiency Diseases 0.000 description 18
- 230000008901 benefit Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 8
- 230000010076 replication Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 239000008186 active pharmaceutical agent Substances 0.000 description 3
- 238000001360 collision-induced dissociation Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000001808 coupling effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000003362 replicative effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000008685 targeting Effects 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a distributed file content management method, a device, a system and a storage medium, and belongs to the technical field of P2P. When a content manager in a content management cluster receives a content operation instruction, determining a content operation type and a target content manager and/or a target node according to the content operation instruction; and performing content operation according to the content operation type and the target content manager and/or the target node to finish the content operation instruction. In this way, a plurality of content managers are configured based on the manner of the content management cluster, and are responsible for different content management operations and distributed content management. The method ensures the rapidness of client development, improves the manageability, adds more functions and improves the usability, thereby further improving the usability of the content and the system.
Description
Technical Field
The present invention relates to the field of P2P technologies, and in particular, to a method, an apparatus, a system, and a storage medium for managing distributed file contents.
Background
The P2P technology is a network information exchange manner, which breaks through the traditional client/server mode, so that each node in the network is peer-to-peer, and can provide services and enjoy services. The P2P distributed file system can effectively utilize a large number of common nodes scattered in the network to store files on different nodes in a scattered mode, and achieves the advantages of high performance, expandability, fault tolerance, decentralization, privacy protection and the like.
There are P2P distributed file systems that employ a fully distributed structured topology type to provide content storage services within an enterprise, which is a DHT and content addressing (Content Addressed) based file system that allows users to store and access data on different nodes, but these products do not provide a convenient way to control on which nodes the data is stored and how to guarantee availability and consistency of the data.
The foregoing is provided merely for the purpose of facilitating understanding of the technical solutions of the present invention and is not intended to represent an admission that the foregoing is prior art.
Disclosure of Invention
The invention mainly aims to provide a distributed file content management method, a distributed file content management device, a distributed file content management system and a distributed file content management storage medium, and aims to solve the technical problem that the availability and consistency of data cannot be guaranteed in the prior art.
In order to achieve the above object, the present invention provides a distributed file content management method, comprising the steps of:
when a content manager in a content management cluster receives a content operation instruction, determining a content operation type and a target content manager and/or a target node according to the content operation instruction;
and performing content operation according to the content operation type and the target content manager and/or the target node to finish the content operation instruction.
Optionally, the determining the content operation type according to the content operation instruction, and the target content manager and/or the target node includes:
when the content operation type is determined to be uploading content, determining an allocation strategy according to the content operation instruction;
and determining a target content manager and a target node for storing the file according to the allocation strategy.
Optionally, the content operation is performed according to the content operation type and the target content manager and/or the target node, so as to complete the content operation instruction, including:
when the content operation type is uploading content, determining a storage file according to the content operation instruction;
storing the storage file into each target node, and obtaining the content identification of the storage file;
and storing the content identification into a locking set through a consistency algorithm component, and notifying each target content manager.
Optionally, the determining the content operation type according to the content operation instruction, and the target content manager and/or the target node includes:
when the content operation type is determined to be copy-locked content, uploading the operation content to a distributed file system according to the content operation instruction;
determining the number of stored copies and the content validity period according to the content operation instruction;
and selecting a target content manager for copying according to the stored copy number and the content validity period.
Optionally, the content operation is performed according to the content operation type and the target content manager and/or the target node, so as to complete the content operation instruction, including:
when the content operation type is copy locking content, adding and modifying a locking set according to the content operation instruction;
the objects in the locked set are synchronized in each target content manager and synchronized to each target content manager upon hearing that there is an object update for the locked set.
Optionally, the content operation is performed according to the content operation type and the target content manager and/or the target node, and after the content operation instruction is completed, the method further includes:
when detecting that the content to be deleted exists, sending an unlocking content identifier to a target unlocking content manager through a client so as to unlock the unlocking content identifier on a node in a locking set through the target unlocking content manager;
and deleting the content to be deleted from each node through a garbage collection mechanism.
Optionally, the content operation is performed according to the content operation type and the target content manager and/or the target node, and after the content operation instruction is completed, the method further includes:
when receiving a reading request content identifier sent by a client, determining a reading node according to the reading request content identifier;
and reading the content data corresponding to the read request content identifier from the reading node.
In addition, to achieve the above object, the present invention also proposes a distributed file content management apparatus including:
the target determining module is used for determining the content operation type and the target content manager and/or the target node according to the content operation instruction when the content manager in the content management cluster receives the content operation instruction;
and the instruction completion module is used for performing content operation according to the content operation type and the target content manager and/or the target node to complete the content operation instruction.
In addition, to achieve the above object, the present invention also proposes a distributed file content management system, including: the system comprises a plurality of nodes, a plurality of clients, a content management cluster and a consistency algorithm component, wherein the content management cluster comprises a plurality of content managers, and the distributed file content management system is used for realizing the distributed file content management method.
In addition, in order to achieve the above object, the present invention also proposes a storage medium having stored thereon a distributed file content management program which, when executed by a processor, implements the distributed file content management method as described above.
When a content manager in a content management cluster receives a content operation instruction, determining a content operation type and a target content manager and/or a target node according to the content operation instruction; and performing content operation according to the content operation type and the target content manager and/or the target node to finish the content operation instruction. In this way, a plurality of content managers are configured based on the manner of the content management cluster, and are responsible for different content management operations and distributed content management. The method ensures the rapidness of client development, improves the manageability, adds more functions and improves the usability, thereby further improving the usability of the content and the system.
Drawings
FIG. 1 is a schematic diagram of a distributed file content management system for a hardware runtime environment in accordance with an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a distributed file content management method according to a first embodiment of the present invention;
FIG. 3 is a schematic diagram of a conventional system structure in an embodiment of a distributed file content management method according to the present invention;
FIG. 4 is a schematic diagram of a system according to an embodiment of the present invention;
FIG. 5 is a flowchart illustrating a distributed file content management method according to a second embodiment of the present invention;
FIG. 6 is a diagram illustrating a copy-locked content flow in an embodiment of a method for managing distributed file content according to the present invention;
fig. 7 is a block diagram illustrating a first embodiment of a distributed file content management apparatus according to the present invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Referring to fig. 1, fig. 1 is a schematic structural diagram of a distributed file content management system of a hardware running environment according to an embodiment of the present invention.
As shown in fig. 1, the distributed file content management system may include: a processor 1001, such as a central processing unit (Central Processing Unit, CPU), a communication bus 1002, a user interface 1003, a network interface 1004, a memory 1005. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a Wireless interface (e.g., a Wireless-Fidelity (Wi-Fi) interface). The Memory 1005 may be a high-speed random access Memory (Random Access Memory, RAM) Memory or a stable nonvolatile Memory (NVM), such as a disk Memory. The memory 1005 may also optionally be a storage device separate from the processor 1001 described above.
Those skilled in the art will appreciate that the architecture shown in fig. 1 is not limiting of a distributed file content management system and may include more or fewer components than shown, or may combine certain components, or may be a different arrangement of components.
As shown in fig. 1, an operating system, a network communication module, a user interface module, and a distributed file content management program may be included in the memory 1005 as one type of storage medium.
In the distributed file content management system shown in fig. 1, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the distributed file content management system of the present invention may be disposed in the distributed file content management system, and the distributed file content management system calls the distributed file content management program stored in the memory 1005 through the processor 1001 and executes the distributed file content management method provided by the embodiment of the present invention.
An embodiment of the present invention provides a distributed file content management method, and referring to fig. 2, fig. 2 is a flowchart of a first embodiment of a distributed file content management method according to the present invention.
In this embodiment, the distributed file content management method includes the following steps:
step S10: when a content manager in the content management cluster receives a content operation instruction, determining a content operation type and a target content manager and/or a target node according to the content operation instruction.
In this embodiment, the execution body of the embodiment may be the distributed file content management system, where the distributed file content management system has functions of data processing, data communication, program running, and the like, and the distributed file content management system may be a centralized controller in the distributed file management system, and may be a server or an intelligent terminal. Of course, other devices with similar functions may be used, and the implementation conditions are not limited thereto. For convenience of explanation, this embodiment will be described by taking a distributed file content management system as an example.
It should be noted that the P2P technology is a network information exchange manner, which breaks through the conventional client/server mode, so that each node in the network is peer-to-peer, and can provide services and enjoy services. The P2P distributed file system can effectively utilize a large number of common nodes scattered in the network to store files on different nodes in a scattered mode, and achieves the advantages of high performance, expandability, fault tolerance, decentralization, privacy protection and the like. The P2P distributed file system has the following types: centralizing topology: this type of system relies on a central server to maintain index and location information for files, such as Napster. The system of this type has the advantages of high discovery efficiency and capability of supporting complex queries, but has the disadvantages of single point failure, performance bottlenecks, legal risks and the like. Full distributed unstructured topology: this type of system does not have a central server, but rather employs a random graph or small world network as the overlay network, e.g., gnutella. This type of system has the advantage of being robust and capable of supporting complex queries, but has the disadvantage of being inefficient in discovery, requiring significant message transmission and storage overhead. Full distributed structured topology: this type of system also does not have a central server, but rather employs a Distributed Hash Table (DHT) as an overlay network, e.g., BT, IPFS. This type of system has the advantage of high discovery efficiency, enabling load balancing and range querying, but has the disadvantage of not supporting complex queries, requiring maintenance of a structured topology. Semi-distributed topology: this type of system combines the centralized and fully distributed features and employs supernode technology to improve efficiency and reliability, such as KaZaA. This type of system has the advantage of high discovery efficiency and can support complex queries, but has the disadvantage that supernodes can be a bottleneck or target of attack. Some enterprises currently employ a P2P distributed file system of the fully distributed structured topology type to provide content storage services within the enterprise, which is a DHT and content addressing (Content Addressed) based file system that allows users to store and access data on different nodes, but these products do not provide a convenient way to control on which nodes the data is stored and how to ensure availability and consistency of the data. To solve the usability and consistency problems, a scheme as shown in fig. 3 is adopted. The scheme is that tasks for solving consistency and usability problems are put at a client, and the working principle is as follows: the client alternately uploads the file contents to the node [1,2. ], or simultaneously uploads a plurality of file contents to the node [1,2. ] through multithreading. Thus, redundancy of files on a plurality of nodes is realized, and the usability problem is solved.
It should be appreciated that the solution shown in fig. 3, while solving the usability problem to some extent, has a great disadvantage: 1. programming is complex/labor intensive. The client application needs to ensure consistency of content across multiple nodes, which is very complex to program and difficult. Most programmers often have difficulty doing safe and reliable, always leak holes and are difficult to maintain. 2. The coupling property is high. If the node is revoked, the node is newly added, the node address is changed, and the client application program needs to be reconfigured, which may also affect the service. 3. Lack of manageability. In order to more conveniently manage and monitor nodes and content in a cluster, such as content lifecycle management, consistency management, node change awareness and response, many very complex tasks are required by application programs, programmers tend to be aware of the difficulty and do not consider the ability to do so. 4. The work is repeated. The same problem is encountered for different applications, but the development has to be repeated. In order to solve these disadvantages, the solution of the present embodiment is proposed.
In a specific implementation, as shown in fig. 4, content manager swarm (content management cluster) is adopted in the scheme to realize internal consistency, manageability and finally improve the usability of the system. The system comprises the following components: 1. node [1,2. ]: and the node of the local P2P system stores the file content uploaded by the client and links with the P2P file system, so that the local file content can be acquired by the whole network. 2. Content manager (Content manager): this is the core of the solution. Its function is to provide data orchestration between nodes [1,2 ], by distributing, copying, and tracking a set (lockset) of global lock files distributed among multiple nodes. It may use different consensus algorithms to coordinate the storage, such as Raft or Merkle-CRDT34. Different data storage backend may also be used, such as bar or Pebble1. It may provide REST APIs via HTTP or libp2p, or may operate via its own command line tools (CLI). 3. Distributed coherency algorithm component (RAFT protocol): the role of using RAFT in Content Manager (CM) is to achieve strong consistency between cluster nodes so that the lockset of all nodes remains consistent and to manage the joining and exiting of nodes. Api: http-APIs are provided on content management, and clients and CLI provide capabilities through API calls content manager swarm. Cli (Command line console) command line client. With which the user can perform more operations, such as: adding, deleting, listing, or restoring Lock files, viewing or modifying cluster configuration, viewing or managing cluster nodes, viewing cluster states or metrics, and viewing or modifying cluster metadata.
Note that RAFT: RAFT is a consistency algorithm in CMS for achieving consistency and replication between CMS. It is responsible for maintaining and replicating the state of Lockset, ensuring that all nodes in the cluster have the same Lockset data, and performing data replication and synchronization when needed. The RAFT algorithm realizes distributed consistency through mechanisms such as leader election, log replication, consistency check and the like.
In an implementation, lockset (lock set): lockset refers to the set of data that is locked (lock) in the CMS, i.e., the file or directory that needs to be persisted and replicated. Lockset may contain multiple CIDs. When a file or directory is added to Lockset, the CM will guarantee Lockset consistency across multiple CM nodes by the RAFT algorithm.
It should be understood that the data structure of Lockset:
wherein, CID: a CID that is lock; expiresat: the effective time of CID; replicas: the number of copies of the content; positioning: it is at which nodes the content referenced by the CID co-negotiated by several CMs in the system is present.
Through the lockset data structure, the state of the lock can be managed among multiple nodes in the CMS, coordinating the lock and unlock operations to ensure consistency. Enabling lock operations to be distributed among multiple CM nodes, the CM may provide a mechanism to track the state and copy condition of a lock object within a cluster.
When any one content manager in the content management cluster receives the content operation instruction, the content operation type is determined according to the content operation instruction, and then the target content manager and/or the target node which need to perform collaborative operation after the operation is completed are determined. After obtaining the content operation instruction, determining the content operation instruction to determine a specific object to be operated, an operation mode and the like, and then determining a content operation type according to the operation mode, wherein the content operation type comprises but is not limited to uploading content, deleting content, copying and locking content, reading content and the like based on the difference of the operation modes. Each collaborative device or instance may include one or more target content managers, at least one of the one or more target nodes, and one.
Further, in order to perform selection of the target content manager and determination of the target node when determining that the content operation type is uploading content, step S10 includes: when the content operation type is determined to be uploading content, determining an allocation strategy according to the content operation instruction; and determining a target content manager and a target node for storing the file according to the allocation strategy.
It should be appreciated that when the content operation type is uploading content, a request for adding a file is sent to a certain CM by the client.
In implementations, the request contains the contents of the file and parameters such as replication factors (several copies), allocation policies (at which nodes copies were made), content expiration dates, etc. And selecting proper nodes to store the file through negotiation according to the replication factors, the distribution strategies and the content validity period between the CM, and obtaining the target node.
Step S20: and performing content operation according to the content operation type and the target content manager and/or the target node to finish the content operation instruction.
After determining the content operation type, the target content manager and/or the target node, the method cooperates with all the target content managers and the target nodes to operate according to different content operation types, so as to complete the distributed management of the file content.
Further, in order to complete the content operation of uploading the content, step S20 includes: when the content operation type is uploading content, determining a storage file according to the content operation instruction; storing the storage file into each target node, and obtaining the content identification of the storage file; and storing the content identification into a locking set through a consistency algorithm component, and notifying each target content manager.
It should be noted that, first, a storage file is determined according to a content operation instruction, then the storage file is stored on one of the nodes [1,2 ], and the CID of the content is obtained, the CID is put into a lock set through a RAFT component, and each CM is notified.
Further, in order to delete the content of the distributed system, after step S20, the method further includes: when detecting that the content to be deleted exists, sending an unlocking content identifier to a target unlocking content manager through a client so as to unlock the unlocking content identifier on a node in a locking set through the target unlocking content manager; and deleting the content to be deleted from each node through a garbage collection mechanism.
It should be appreciated that when the content is no longer needed, the client may send an unlock content identification (unlock CID) request CM to unlock the CID, which may unlock the CID on the node according to the state of the lock set maintained by the RAFT protocol component, and the node may then delete the content from the local node by a garbage collection mechanism (GC) when appropriate. If the content validity period parameter is specified when uploading the content, the CM automatically sends an unlock CID to the node.
In this way, the lifecycle of the content can be managed automatically. An expiration time attribute is added to each CID content, and content manager automatically unlocks an expiration file according to the expiration time. And when a certain node is replaced, the configuration can be modified in the content manager, and the client application does not need to change the configuration.
Further, in order to facilitate the other devices to read the file contents in the system, after step S20, the method further includes: when receiving a reading request content identifier sent by a client, determining a reading node according to the reading request content identifier; and reading the content data corresponding to the read request content identifier from the reading node.
In implementations, other clients may send read requests (get CIDs) to read the CIDs' content from any node in the P2P distributed file system.
When a content manager in a content management cluster receives a content operation instruction, determining a content operation type and a target content manager and/or a target node according to the content operation instruction; and performing content operation according to the content operation type and the target content manager and/or the target node to finish the content operation instruction. In this way, a plurality of content managers are configured based on the manner of the content management cluster, and are responsible for different content management operations and distributed content management. The method ensures the rapidness of client development, improves the manageability, adds more functions and improves the usability, thereby further improving the usability of the content and the system.
Referring to fig. 5, fig. 5 is a flowchart illustrating a second embodiment of a distributed file content management method according to the present invention.
Based on the first embodiment, the distributed file content management method of the present embodiment includes, at step S10:
step S101: and when the content operation type is determined to be copy-locked content, uploading the operation content to a distributed file system according to the content operation instruction.
It should be noted that, as shown in fig. 6, which is an operation diagram of copying the locked content, after the CM receives the CID, if it finds that it is assigned with the task of storing the file, it sends a lock CID request to the node, and stores the file content to the corresponding node. At this point, the CID content is stored by the number of nodes specified by the replication factor. After the CM completes the lock operation, the CM updates its own state and synchronizes to other nodes via the consensus component. If the lock operation fails or times out, recovery is attempted periodically.
Step S102: and determining the number of the stored copies and the content validity period according to the content operation instruction.
It should be appreciated that the content is uploaded to the CM1 node and the validity period and number of copies of the content (assuming 2 is possible) are notified.
Step S103: and selecting a target content manager for copying according to the stored copy number and the content validity period.
In an implementation, as shown in fig. 6, the CM1 stores the content on a node of the distributed file system and obtains the CID of the file; CM1 negotiates with CM2, CM3, and selects nodes to store and lock content according to the number of copies.
Further, in order to copy-lock content after determining the target content manager, according to the content operation type and the content operation performed by the target content manager and/or the target node, the step of completing the content operation instruction includes: when the content operation type is copy locking content, adding and modifying a locking set according to the content operation instruction; the objects in the locked set are synchronized in each target content manager and synchronized to each target content manager upon hearing that there is an object update for the locked set.
The CM1 and CM3 nodes are selected through negotiation; adding and modifying a lockset object; the lock set objects are synchronized among the CM1, the CM2 and the CM3 through the RAFT protocol component, so that each CM node is guaranteed to have the same lock set object. The CM1 and the CM3 monitor that new content in the lock set object needs the local node to copy and lock, and then perform lock to lock the content copy on the distributed file system nodes of the CM1 and the CM 3.
In this cooperative manner, lockset and RAFT ensure data consistency, high availability, and data replication between nodes in the CMS, thereby providing reliable distributed file system management capabilities.
When the content operation type is determined to be copy-locked content, uploading the operation content to a distributed file system according to the content operation instruction; determining the number of stored copies and the content validity period according to the content operation instruction; and selecting a target content manager for copying according to the stored copy number and the content validity period. In this way, it is achieved that the state of a lock can be managed among multiple nodes in the CMS, coordinating lock and unlock operations to ensure consistency, through a lock set data structure. Enabling lock operations to be distributed among multiple CM nodes, the CM may provide a mechanism to track the state and copy condition of a lock object within a cluster.
In addition, the embodiment of the invention also provides a storage medium, wherein the storage medium is stored with a distributed file content management program, and the distributed file content management program realizes the steps of the distributed file content management method when being executed by a processor.
Because the storage medium adopts all the technical schemes of all the embodiments, the storage medium has at least all the beneficial effects brought by the technical schemes of the embodiments, and the description is omitted here.
Referring to fig. 7, fig. 7 is a block diagram showing the structure of a first embodiment of the distributed file content managing apparatus according to the present invention.
As shown in fig. 7, the distributed file content management apparatus according to the embodiment of the present invention includes:
the target determining module 10 is configured to determine a content operation type and a target content manager and/or a target node according to a content operation instruction when the content manager in the content management cluster receives the content operation instruction.
And the instruction completion module 20 is configured to perform content operations according to the content operation type and the target content manager and/or the target node, so as to complete the content operation instruction.
When a content manager in a content management cluster receives a content operation instruction, determining a content operation type and a target content manager and/or a target node according to the content operation instruction; and performing content operation according to the content operation type and the target content manager and/or the target node to finish the content operation instruction. In this way, a plurality of content managers are configured based on the manner of the content management cluster, and are responsible for different content management operations and distributed content management. The method ensures the rapidness of client development, improves the manageability, adds more functions and improves the usability, thereby further improving the usability of the content and the system.
In an embodiment, the objective determining module 10 is further configured to determine, when determining that the content operation type is uploading content, an allocation policy according to the content operation instruction;
and determining a target content manager and a target node for storing the file according to the allocation strategy.
In an embodiment, the instruction completing module 20 is further configured to determine, when the content operation type is uploading content, a storage file according to the content operation instruction; storing the storage file into each target node, and obtaining the content identification of the storage file; and storing the content identification into a locking set through a consistency algorithm component, and notifying each target content manager.
In an embodiment, the targeting module 10 is further configured to upload the operation content to the distributed file system according to the content operation instruction when determining that the content operation type is copy-locked content; determining the number of stored copies and the content validity period according to the content operation instruction; and selecting a target content manager for copying according to the stored copy number and the content validity period.
In an embodiment, the instruction completing module 20 is further configured to add and modify a lock set according to the content operation instruction when the content operation type is copy-locked content; the objects in the locked set are synchronized in each target content manager and synchronized to each target content manager upon hearing that there is an object update for the locked set.
In an embodiment, the instruction completing module 20 is further configured to send, when it is detected that there is content to be deleted, an unlock content identifier to a target unlock content manager through the client, so as to unlock, by the target unlock content manager, the unlock content identifier on the node in the lock set; and deleting the content to be deleted from each node through a garbage collection mechanism.
In an embodiment, the instruction completing module 20 is further configured to, when receiving a read request content identifier sent by a client, determine a read node according to the read request content identifier; and reading the content data corresponding to the read request content identifier from the reading node.
It should be understood that the foregoing is illustrative only and is not limiting, and that in specific applications, those skilled in the art may set the invention as desired, and the invention is not limited thereto.
It should be noted that the above-described working procedure is merely illustrative, and does not limit the scope of the present invention, and in practical application, a person skilled in the art may select part or all of them according to actual needs to achieve the purpose of the embodiment, which is not limited herein.
In addition, technical details that are not described in detail in this embodiment may refer to the distributed file content management method provided in any embodiment of the present invention, which is not described herein.
Furthermore, it should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. Read Only Memory)/RAM, magnetic disk, optical disk) and including several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.
Claims (10)
1. A distributed file content management method, the distributed file content management method comprising:
when a content manager in a content management cluster receives a content operation instruction, determining a content operation type and a target content manager and/or a target node according to the content operation instruction;
and performing content operation according to the content operation type and the target content manager and/or the target node to finish the content operation instruction.
2. The method for managing distributed file contents according to claim 1, wherein when the content manager in the content management cluster receives the content operation instruction, determining the content operation type according to the content operation instruction, and the target content manager and/or the target node, comprises:
when the content operation type is determined to be uploading content, determining an allocation strategy according to the content operation instruction;
and determining a target content manager and a target node for storing the file according to the allocation strategy.
3. The method for managing distributed file contents according to claim 2, wherein said performing a content operation according to said content operation type and said target content manager and/or target node, completing said content operation instruction, comprises:
when the content operation type is uploading content, determining a storage file according to the content operation instruction;
storing the storage file into each target node, and obtaining the content identification of the storage file;
and storing the content identification into a locking set through a consistency algorithm component, and notifying each target content manager.
4. The distributed file content management method according to claim 1, wherein said determining a content operation type according to said content operation instruction, and a target content manager and/or a target node, comprises:
when the content operation type is determined to be copy-locked content, uploading the operation content to a distributed file system according to the content operation instruction;
determining the number of stored copies and the content validity period according to the content operation instruction;
and selecting a target content manager for copying according to the stored copy number and the content validity period.
5. The method for managing distributed file contents according to claim 4, wherein said performing a content operation according to said content operation type and said target content manager and/or target node, completing said content operation instruction, comprises:
when the content operation type is copy locking content, adding and modifying a locking set according to the content operation instruction;
the objects in the locked set are synchronized in each target content manager and synchronized to each target content manager upon hearing that there is an object update for the locked set.
6. A distributed file content management method according to any one of claims 1 to 5, wherein said performing a content operation according to said content operation type and said target content manager and/or target node, after completing said content operation instruction, further comprises:
when detecting that the content to be deleted exists, sending an unlocking content identifier to a target unlocking content manager through a client so as to unlock the unlocking content identifier on a node in a locking set through the target unlocking content manager;
and deleting the content to be deleted from each node through a garbage collection mechanism.
7. A distributed file content management method according to any one of claims 1 to 5, wherein said performing a content operation according to said content operation type and said target content manager and/or target node, after completing said content operation instruction, further comprises:
when receiving a reading request content identifier sent by a client, determining a reading node according to the reading request content identifier;
and reading the content data corresponding to the read request content identifier from the reading node.
8. A distributed file content management apparatus, the distributed file content management apparatus comprising:
the target determining module is used for determining the content operation type and the target content manager and/or the target node according to the content operation instruction when the content manager in the content management cluster receives the content operation instruction;
and the instruction completion module is used for performing content operation according to the content operation type and the target content manager and/or the target node to complete the content operation instruction.
9. A distributed file content management system, the distributed file content management system comprising: a plurality of nodes, a plurality of clients, a content management cluster comprising a plurality of content managers, and a consistency algorithm component, the distributed file content management system for implementing the distributed file content management method of claims 1-7.
10. A storage medium having stored thereon a distributed file content management program which when executed by a processor implements the distributed file content management method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410024966.2A CN117785800A (en) | 2024-01-03 | 2024-01-03 | Distributed file content management method, device, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410024966.2A CN117785800A (en) | 2024-01-03 | 2024-01-03 | Distributed file content management method, device, system and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117785800A true CN117785800A (en) | 2024-03-29 |
Family
ID=90396208
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410024966.2A Pending CN117785800A (en) | 2024-01-03 | 2024-01-03 | Distributed file content management method, device, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117785800A (en) |
-
2024
- 2024-01-03 CN CN202410024966.2A patent/CN117785800A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8195742B2 (en) | Distributed client services based on execution of service attributes and data attributes by multiple nodes in resource groups | |
| Hasan et al. | A survey of peer-to-peer storage techniques for distributed file systems | |
| JP6628730B2 (en) | Geographically distributed file system using equivalent namespace replicas over wide area networks | |
| Goel et al. | Data replication strategies in wide-area distributed systems | |
| US9053167B1 (en) | Storage device selection for database partition replicas | |
| US7634566B2 (en) | Arrangement in a network for passing control of distributed data between network nodes for optimized client access based on locality | |
| JP4538454B2 (en) | Search for electronic document replicas in computer networks | |
| US20100161657A1 (en) | Metadata server and metadata management method | |
| JP6059216B2 (en) | Method and apparatus for distributed configuration management | |
| JP7270755B2 (en) | Metadata routing in distributed systems | |
| JP2003248611A (en) | Storage management integration system and its storage management control method | |
| Rajalakshmi et al. | An improved dynamic data replica selection and placement in cloud | |
| Dustdar et al. | Dynamic replication and synchronization of web services for high availability in mobile ad-hoc networks | |
| Xhafa et al. | Data replication in P2P collaborative systems | |
| Akbarinia et al. | Data management in the APPA system | |
| Duan et al. | Two-layer hybrid peer-to-peer networks | |
| CN107547657A (en) | A kind of method, apparatus and storage medium numbered based on one point data in cloud storage system | |
| Richard et al. | Clique: A transparent, peer-to-peer replicated file system | |
| CN117785800A (en) | Distributed file content management method, device, system and storage medium | |
| Pérez-Miguel et al. | High throughput computing over peer-to-peer networks | |
| van Renesse et al. | Autonomic computing: A system-wide perspective | |
| JP3835199B2 (en) | Distributed management network file system and file method | |
| Martins et al. | Scalable and topology-aware reconciliation on P2P networks | |
| Chazapis et al. | Xoros: A mutable distributed hash table | |
| Frolund et al. | Scalable state replication with weak consistency |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |