CN116756101A - Storage distributed cleaning method and device based on cloud rendering - Google Patents

Storage distributed cleaning method and device based on cloud rendering Download PDF

Info

Publication number
CN116756101A
CN116756101A CN202310666126.1A CN202310666126A CN116756101A CN 116756101 A CN116756101 A CN 116756101A CN 202310666126 A CN202310666126 A CN 202310666126A CN 116756101 A CN116756101 A CN 116756101A
Authority
CN
China
Prior art keywords
cleaning
storage
storage area
instruction
user
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310666126.1A
Other languages
Chinese (zh)
Inventor
徐俊杰
高斌
邹琼
周双全
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Ruiyun Technology Co ltd
Original Assignee
Shenzhen Ruiyun Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Ruiyun Technology Co ltd filed Critical Shenzhen Ruiyun Technology Co ltd
Priority to CN202310666126.1A priority Critical patent/CN116756101A/en
Publication of CN116756101A publication Critical patent/CN116756101A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/16File or folder operations, e.g. details of user interfaces specifically adapted to file systems
    • G06F16/162Delete operations
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/17Details of further file system functions
    • G06F16/1727Details of free space management performed by the file system
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/17Details of further file system functions
    • G06F16/174Redundancy elimination performed by the file system
    • G06F16/1748De-duplication implemented within the file system, e.g. based on file segments

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention relates to a storage distributed cleaning method and device based on cloud rendering, wherein the method comprises the following steps: acquiring a clearing instruction of a user, acquiring storage data of each storage area through a clearing timer based on the clearing instruction of the user, and analyzing and generating execution parameters corresponding to each storage area; wherein each storage area is provided with an independent storage server; the execution parameters include the storage path, file type and file expiration time; judging a target storage area to be cleaned through a cleaning timer based on the execution parameters and the cleaning instruction; obtaining a scanning result obtained by a storage server of a target storage area, wherein the scanning result is obtained based on the clearing instruction and the execution parameter; and cleaning the scanning result of the target storage area to solve the problem of low storage space utilization rate.

Description

Storage distributed cleaning method and device based on cloud rendering
Technical Field
The invention belongs to the technical field of cloud computing, and particularly relates to a storage distributed cleaning method and device based on cloud rendering.
Background
The cloud rendering platform is a cloud computing service and can provide efficient rendering services for users. According to the method, the rendering speed can be greatly improved by distributing the rendering tasks to a large amount of computing resources of the cloud, so that a user can complete complex rendering tasks more quickly. Cloud rendering platforms typically provide support for a variety of rendering software, including 3ds Max, maya, blender, and so forth, common 3D rendering software. The user can upload the 3D model and the scene file to the cloud, and then configure and submit tasks through an interface provided by the cloud rendering platform. The platform can automatically allocate computing resources and return results to the user after rendering is completed, and has the remarkable advantages of high efficiency, expandability, high cost efficiency, convenience and the like.
In cloud rendering tasks, storage management is very important. Cloud rendering tasks require a large amount of data storage, such as rendering scenes, textures, maps, etc., which tend to be very large. If there is insufficient storage space, the data cannot be stored, resulting in rendering tasks being disabled. In the rendering task, different data needs to be processed and managed separately, otherwise, data confusion and errors occur. Therefore, there is a need for an efficient data management system that classifies, cleans up and backs up data for better rendering tasks.
The existing cloud rendering platform has the defect of low storage clearance function, so that the problem of low storage space utilization rate is caused.
Disclosure of Invention
In view of the above, a storage distributed cleaning method and device based on cloud rendering are provided to solve the problem of low storage space utilization in the prior art.
The invention adopts the following technical scheme:
in a first aspect, an embodiment of the present invention provides a storage distributed cleaning method based on cloud rendering, where the method includes:
acquiring a clearing instruction of a user, wherein the clearing instruction comprises a customized instruction; the customization instructions comprise customization strategies of how the stored data are cleaned by a user, wherein the customization strategies comprise file cleaning periods, file types and storage paths which are set by the user;
based on a cleaning instruction of a user, acquiring storage data of each storage area through a cleaning timer, and analyzing and generating execution parameters corresponding to each storage area; wherein each storage area is provided with an independent storage server; the execution parameters include the storage path, file type and file expiration time;
judging a target storage area to be cleaned through a cleaning timer based on the execution parameters and the cleaning instruction;
obtaining a scanning result obtained by a storage server of a target storage area, wherein the scanning result is obtained based on the clearing instruction and the execution parameter;
and cleaning the scanning result of the target storage area.
Further, the clearing instruction further comprises a preset default clearing rule of the system.
Further, the cleaning the scan result of the target storage area includes:
acquiring and analyzing the scanning result through a cleaning timer, and generating job information of a user, wherein the job information represents the specific completion time of the current user of the current job; scanning the scanning result based on the operation information and the cleaning instruction through a cleaning timer to generate a cleaning list, and caching the cleaning list;
acquiring the cleaning list of the target storage area, and backing up the cleaning list;
generating a folder of a corresponding recycle bin according to the current date from the cleaning list, wherein the folder is stored in a recycle bin directory;
and calling a storage server of the target storage area to clean up the backup, and deleting the recycle bin directory.
Further, the method further comprises the following steps:
and deleting the recycle bin catalogue according to a preset period.
Further, the obtaining the scan result obtained by the storage server of the target storage area, where the scan result is obtained based on the purge instruction and the execution parameter, includes:
and if the scanning result is empty, acquiring the last scanning result.
Further, the method comprises the steps of:
and caching the scanning result.
Further the storage area comprises a complete type and a storage server;
each storage area is used for executing different rendering tasks;
the complete type representation at least comprises one of each type of stored data required for completing the rendering task;
and the storage areas are isolated from each other.
Further, after the storage server for obtaining the target storage area and before the scan result obtained based on the purge instruction and the execution parameter, the method further includes:
acquiring storage data of a target storage area, judging whether the storage data is an active user, if so, inquiring the ID of the active user, and determining the expiration time of the user;
and if not, screening based on the clearing instruction and the execution parameters to obtain a scanning result.
In a second aspect, an embodiment of the present invention provides a storage distributed cleaning device based on cloud rendering, where the storage distributed cleaning device includes:
the system comprises an acquisition module, a control module and a control module, wherein the acquisition module is used for acquiring a clearing instruction of a user, and the clearing instruction comprises a customization instruction; the customization instructions comprise customization strategies of how the stored data are cleaned by a user, wherein the customization strategies comprise file cleaning periods, file types and storage paths which are set by the user;
the distributed storage and analysis module is used for acquiring the storage data of each storage area through a cleaning timer based on a cleaning instruction of a user, and analyzing and generating execution parameters corresponding to each storage area; wherein each storage area is provided with an independent storage server; the execution parameters include the storage path, file type and file expiration time;
the judging module is used for judging a target storage area to be cleaned through a cleaning timer based on the execution parameters and the cleaning instruction;
the scheduling module is used for acquiring a scanning result obtained by a storage server of the target storage area, wherein the scanning result is obtained based on the clearing instruction and the execution parameters;
and the cleaning module is used for cleaning the scanning result of the target storage area.
Drawings
In order to more clearly illustrate the embodiments of the invention or the solutions of the prior art, the drawings which are required in the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the description below are only some embodiments of the invention and that other drawings may be obtained from these drawings by a person skilled in the art without the benefit of the inventive effort.
FIG. 1 is a flowchart of a storage distributed cleaning method based on cloud rendering according to an embodiment of the present invention;
FIG. 2 is a schematic block diagram of a storage distributed cleaning device based on cloud rendering according to an embodiment of the present invention;
FIG. 3 is a flowchart of a background cleaning implementation of a storage distributed cleaning device based on cloud rendering according to another embodiment of the present invention;
fig. 4 is an application flowchart of a storage distributed cleaning method based on cloud rendering according to another embodiment of the present invention.
Detailed Description
The following describes the present invention in detail in order to further clarify the objects, technical solutions and advantages of the present invention. It will be apparent that the described embodiments are only some, but not all, embodiments of the invention. All other embodiments, based on the examples herein, which are within the scope of the invention as defined by the claims, will be within the scope of the invention as defined by the claims.
Example 1
Fig. 1 is a flowchart of a storage distributed cleaning method based on cloud rendering according to an embodiment of the present invention, and referring to fig. 1, the method may specifically include the following steps:
s11, step: acquiring a clearing instruction of a user, wherein the clearing instruction comprises a customized instruction; the customization instructions comprise customization strategies of how the stored data are cleaned by a user, wherein the customization strategies comprise file cleaning periods, file types and storage paths which are set by the user.
S12, step: based on a cleaning instruction of a user, acquiring storage data of each storage area through a cleaning timer, and analyzing and generating execution parameters corresponding to each storage area; wherein each storage area is provided with an independent storage server; the execution parameters include the storage path, file type, and file expiration time.
S13, step: and judging the target storage area to be cleaned through a cleaning timer based on the execution parameters and the cleaning instruction.
S14, step: and acquiring a scanning result obtained by a storage server of the target storage area, wherein the scanning result is obtained based on the clearing instruction and the execution parameters.
S15, step: and cleaning the scanning result of the target storage area.
Specifically, in step S11, a user' S clearing instruction is obtained, where the clearing instruction includes a custom instruction; the customization instructions comprise a customization strategy of how the user cleans the stored data, wherein the customization strategy comprises a file cleaning period, a file type and a storage path which are set by the user, and the customization strategy comprises the following steps: ,
preferably, the clearing instruction further comprises a preset default clearing rule of the system.
It is worth to say that the clearing instruction includes two kinds of default clearing rules and custom instructions, if no custom instruction is executed directly according to the default of the system, the custom instruction includes a clearing period preset by the user, for example, the clearing period may be one week or five days, that is, clearing is performed once a week or once five days; the method can also clean according to file types and file storage paths, help users to select in a diversified mode, and in practical application, the users can select a proper customization strategy to clean according to own needs so as to better manage storage space.
In step S12, based on a user cleaning instruction, acquiring storage data of each storage area through a cleaning timer, and analyzing and generating execution parameters corresponding to each storage area; wherein each storage area is provided with an independent storage server; the execution parameters include the storage path, file type and file expiration time, which are specifically:
the storage space is divided into a plurality of storage areas, and each storage area respectively manages different types of data. For example, different storage data are respectively stored in different storage areas according to the types of the storage data such as rendering scenes, textures, maps, configuration files and the like, and the storage space can be divided into different platforms according to parameters such as hardware configuration, rendering types and the like. Through storage distributed management, data can be managed more effectively, and the utilization rate of the data is improved.
Further, storage types may also be differentiated according to file types, such as input storage for storing user asset files, output storage for storing user rendering task output files, config storage for storing user configuration files, and so forth. And then dividing different storage areas (i.e. platforms) according to specific rendering services, wherein each platform is required to be provided with at least one set of complete type of storage to support the normal operation of rendering tasks, the storage between the platforms is mutually isolated, various types of storage under the same platform are respectively mounted on different paths on the same linxu server, mounting paths (such as:/mnt/storage type/storage name) which are clear in configuration level and easy to distinguish are recorded in the background, and file indexes in storage management are facilitated. Meanwhile, in order to facilitate the storage server to analyze the file, the stored directory name should also contain part of the file related information, for example, the directory structure of the user ID/task ID is used, so that the storage server can quickly index and analyze the file information.
The values illustrate that the timing cleaner is deployed separately outside the respective storage server, and its main responsibility is the implementation of the scheduling and customization services of the storage service, and through interaction with the distributed storage service, indirect access to the respective storage area (i.e., platform) is achieved. Meanwhile, the judgment of business logic such as storage paths, file types, file expiration time and the like in the user cleaning instructions and execution parameters can be realized by accessing data such as users, rendering jobs, platform configuration, storage paths and the like of the rendering platform.
It will be appreciated that the purge timer indirectly accesses each storage area (i.e., platform), parses stored data within the platform to generate execution parameters, including: storage path, file type and file expiration time.
In step S13, based on the execution parameters and the purge instruction, the target storage area to be purged is judged by the purge timer,
the value is stated that, according to the storage path analyzed by the timing cleaner, the file type, the file expiration time and the cleaning instruction of the user, whether each storage area is required to be cleaned or not is judged, and if so, the storage area is set as the target storage area.
In step S14, "obtaining a scan result obtained by the storage server of the target storage area, where the scan result is obtained based on the purge instruction and the execution parameter":
specifically, the RPC invokes the storage server of each storage area to perform scanning of the stored data, traverses the file sub-paths under each storage root path, screens the files meeting the conditions in the clearing instruction and the execution parameters, generates a scanning result, and caches the scanning result.
It can be understood that, for a determined target storage area, a storage path in storage data is scanned in a targeted manner based on a custom instruction or a default system to be cleaned in a cleaning instruction, a file type and a file expiration time are found to deeply clean a root directory in the storage path, wherein the judgment of the expiration time of the file is herein the expiration judgment of the modification time of the file, the modification time is the system modification time of the file on storage, the expiration time is also a preset value, for example, cleaning is performed once more than 20 days, and a user can set different expiration times according to conditions such as a platform and a storage type. Through the description of the characters, the stored data of each storage area is indirectly accessed through the cleaning timer to realize asynchronous execution and automatic cleaning functions, so that the efficiency of file cleaning is improved, and the complexity and inconvenience of manual cleaning operation are reduced.
In the step S15, "cleaning the scan result of the target storage area", the method specifically includes:
acquiring and analyzing the scanning result through a cleaning timer, and generating job information of a user, wherein the job information represents the specific completion time of the current user of the current job; scanning the scanning result based on the operation information and the cleaning instruction through a cleaning timer to generate a cleaning list, and caching the cleaning list;
acquiring the cleaning list of the target storage area, and backing up the cleaning list;
generating a folder of a corresponding recycle bin according to the current date from the cleaning list, wherein the folder is stored in a recycle bin directory;
and calling a storage server of the target storage area to clean up the backup, and deleting the recycle bin directory.
The value is stated that, the cleaning timer scans the scanning result based on the operation information and the cleaning instruction to generate a cleaning list, the scanning is the completion time of the file, the completion time is the rendering service, the rendering service comprises textures, maps and the like, each file corresponds to the operation rendered on one storage area, the operation is a rendering service unit provided by the cloud rendering platform, the platform records the actual completion time of the operation, namely the operation information, so that the cleaning list generated by scanning the scanning result again based on the operation information and the cleaning instruction is more consistent with the actual situation and more accurate.
It should be noted that, because the cleaning operation is performed, the cleaning operation is not directly performed, and is performed after backup, the cleaning operation is equivalent to that the files deleted on windows are put in the recycle bin for a period of time and then cleaned, and the files can be recovered at any time.
Illustratively, for example, the recycle bin folder is/output/File one, the clean list is/output/2023-5-24/output/File one; the/expire/2023-5-24 is to generate the recycle bin path according to the current date, so that the operation record when cleaning is performed can be kept, and the operation record is convenient to check and trace.
Preferably, the method further comprises:
and deleting the recycle bin catalogue according to a preset period.
For example, if the preset period is 7 days, the recycle bin cleans the block each time it is determined whether there is a path outside seven days, and folders outside seven days are directly deleted.
Preferably, the obtaining the scan result obtained by the storage server of the target storage area, where the scan result is obtained based on the purge instruction and the execution parameter includes:
and if the scanning result is empty, acquiring the last scanning result.
It will be appreciated that a determination is made as to whether the scan result is null, where two scans are performed. The first time is the condition that the scanning result is empty, the file scanning is executed first, then the scanning result is cached, the second time is the scanning result is not empty, the last time is taken, and then the cleaning operation is prepared to be executed.
Preferably, the storage area comprises a complete type and a storage server;
each storage area is used for executing different rendering tasks;
the complete type representation at least comprises one of each type of stored data required for completing the rendering task;
and the storage areas are isolated from each other.
Preferably, after the storage server for obtaining the target storage area and before the scan result obtained based on the purge instruction and the execution parameter, the method further includes:
acquiring storage data of a target storage area, judging whether the storage data is an active user, if so, inquiring the ID of the active user, and determining the expiration time of the user;
and if not, screening based on the clearing instruction and the execution parameters to obtain a scanning result.
The beneficial effects are that: the invention obtains the storage data of each storage area through the cleaning timer based on the cleaning instruction of the user, analyzes and generates the execution parameters corresponding to each storage area, wherein the storage area is divided into different storage areas, and each storage area is provided with an independent storage server, so that the division and the deployment of different storage areas of the storage data are realized, the distributed storage management is completed, and the laying is made for improving the utilization rate of the storage space. Further, when the user issues the cleaning instruction, firstly, judging according to the acquired file type and file expiration time which are analyzed by the cleaning timer and the execution parameters of the storage path, determining one or more target storage areas to be cleaned, then, scanning the storage areas by combining with a customized strategy, namely, a file cleaning period, a file type and a storage path which are set by the user, screening out storage data which meet the execution parameters and the cleaning instruction, generating a scanning result, and cleaning the scanning result to complete the whole automatic cleaning process.
Therefore, a scanning result is generated by accessing the storage server of the target storage area through the cleaning timer based on the cleaning instruction and the execution parameters of the user, and cleaning work is carried out on the scanning structure.
Referring to fig. 3, fig. 3 is a schematic block diagram of a storage distributed cleaning apparatus based on cloud rendering, as shown in the figure, including:
the system comprises an acquisition module 1, a control module and a control module, wherein the acquisition module is used for acquiring a clearing instruction of a user, and the clearing instruction comprises a customization instruction; the customization instructions comprise customization strategies of how the stored data are cleaned by a user, wherein the customization strategies comprise file cleaning periods, file types and storage paths which are set by the user;
the distributed storage and analysis module 2 is used for acquiring the storage data of each storage area through a cleaning timer based on a cleaning instruction of a user, and analyzing and generating execution parameters corresponding to each storage area; wherein each storage area is provided with an independent storage server; the execution parameters include the storage path, file type and file expiration time;
the judging module 3 is used for judging the target storage area to be cleaned through a cleaning timer based on the execution parameters and the cleaning instruction;
the scheduling module 4 is used for acquiring a scanning result obtained by a storage server of the target storage area, wherein the scanning result is obtained based on the clearing instruction and the execution parameter;
and the cleaning module 5 is used for cleaning the scanning result of the target storage area.
Specifically, a storage distributed cleaning device based on cloud rendering may refer to a specific implementation manner of a storage distributed cleaning method based on cloud rendering described in any of the foregoing embodiments, which is not described herein again.
It can be understood that the invention obtains the storage data of each storage area through the cleaning timer based on the cleaning instruction of the user, analyzes and generates the execution parameters corresponding to each storage area, wherein the invention divides the storage area into different storage areas, and each storage area is provided with an independent storage server, thereby realizing the division and deployment of different storage areas of the storage data, completing the distributed storage management and laying a cushion for improving the utilization rate of the storage space. Further, when the user issues the cleaning instruction, firstly, judging according to the acquired file type and file expiration time which are analyzed by the cleaning timer and the execution parameters of the storage path, determining one or more target storage areas to be cleaned, then, scanning the storage areas by combining with a customized strategy, namely, a file cleaning period, a file type and a storage path which are set by the user, screening out storage data which meet the execution parameters and the cleaning instruction, generating a scanning result, and cleaning the scanning result to complete the whole automatic cleaning process.
Therefore, a scanning result is generated by accessing the storage server of the target storage area through the cleaning timer based on the cleaning instruction and the execution parameters of the user, and cleaning work is carried out on the scanning structure.
Example two
Referring to fig. 3, fig. 3 is a flowchart of a background cleaning execution of a storage distributed cleaning apparatus based on cloud rendering, as shown in the drawing, including:
s101: storage space type and region division deployment
S102: deploying a background distributed storage service
S103: deployment background cleaning timer
S104: deployment front end control page
In specific practice, S101: the storage space type and region division deployment comprises the following steps:
storage types are distinguished according to file types, such as input storage for storing user asset files, output storage for storing user rendering task output files, config storage for storing user configuration files, and the like. And then different storage areas (i.e. platforms) are divided according to specific services, each platform needs to be provided with at least one set of complete type of storage to support the normal operation of rendering tasks, the storage among the platforms is mutually isolated, various types of storage under the same platform are respectively mounted on different paths on the same linxu server, mounting paths (such as:/mnt/storage type/storage name) which are clear in configuration level and easy to distinguish are recorded in the background, and file indexes in storage management are facilitated. Meanwhile, in order to facilitate the storage service to analyze the file, the stored directory name should also contain partial file related information, for example, a directory structure using user ID/task ID, so that the storage service can quickly index and analyze the file information.
S102: deploying a background distributed storage service, comprising:
the background storage service is deployed on the storage servers, the storage servers of different platforms are mutually isolated, each platform needs to be deployed respectively, the storage service of distributed deployment can be directly accessed to the storage mounted on the platform server through a linux file system respectively, and the background storage service is an executor of the distributed cleaning module.
S103: deploying a background clean-up timer, comprising:
the background cleaning timer is independently deployed outside the storage server and is the core of the distributed storage cleaning module, and the main responsibility of the background cleaning timer is the realization of the scheduling and customization business of the storage service, and the indirect access to the platform storage is realized through the interaction with the distributed storage service. Meanwhile, business logic such as user customization strategy customization, rendering task file expiration time judgment, active user judgment and the like can be realized by accessing data such as a user of a rendering platform, rendering jobs, platform configuration, storage paths and the like. The purge timer interacts with the storage service through RPC calls. The scans are performed asynchronously, with the result of each scan being a trigger clean by the next timer.
S104: deploying a front-end control page, comprising:
the functions to be implemented include control of the cleaning timer, configuration of the user-customized policy, file restoration, etc., and may be designed according to specific services, which will not be repeated here.
Referring to fig. 4, fig. 4 is an application flowchart of a storage distributed cleaning method based on cloud rendering, as shown in the drawing, including:
analyzing the execution parameters of the stored data through a timer, namely a cleaning timer, judging which of all platforms, namely storage areas, are the stored data to be cleaned by a user through the execution parameters and a cleaning instruction of the user, determining which of the stored data is on which of the platforms, determining that the data is to be cleaned is a target platform, judging whether the stored data on the target platform is only cleaned by an active user or not, if yes, inquiring an active user ID and caching the data, and if not, inquiring expiration time of a file on the target platform and custom data of the user to generate a scanning result, wherein the judgment of the expiration time of the file is the expiration judgment of the modification time of the file, the modification time is the system modification time of the file on the storage, the expiration time is a preset value, for example, cleaning once in more than 20 days, and setting different expiration times by the user according to the conditions of the platform, the storage type and the like; the custom data of the user, namely the custom strategy, has the advantage that the storage data meeting the requirements is screened by the custom strategy to generate the scanning result.
It should be noted that, if the scan result is null, the last scan result is obtained.
It will be appreciated that a determination is made as to whether the scan result is null, where two scans are performed. The first time is the condition that the scanning result is empty, the file scanning is executed first and then the scanning result is cached, wherein the file scanning process is to execute the file scanning by calling the storage server of each platform through the RPC, traverse the file sub-paths under each storage root path, screen the files meeting the conditions of expiration time, active users, customization strategies and the like, generate the scanning result and cache. It should be noted that, if the user does not customize the policy, the screening scan is directly performed according to the default cleaning rule of the system.
The second time the scan result is not empty, the result of the last scan is taken, and then the cleaning operation is ready to be performed.
After the scanning result is obtained, user and job ID information in a file path is analyzed, the user and job ID information refers to which user finishes which job at which time, whether the user is a customized user or not is judged, that is, whether the user has a customized policy or not is judged, if yes, the user customized policy is obtained to clear storage data on a target platform, the scanning analysis is conducted on the fact that the corresponding jobs of the files are all at the end time of a database, namely the completion time, of the corresponding jobs of the files, the rendering service comprises textures, a map and the like, each file corresponds to the rendered jobs on one storage area, the jobs are rendering service units provided by a cloud rendering platform, the platform records the actual completion time of the jobs, the end time of the corresponding jobs of the files in the database is obtained, and a clearing list is screened out and cached.
Furthermore, the storage server of the system calls the storage servers of each platform to execute file backup of the cleaning list through the RPC, the backup list is obtained from the cache, each storage server is traversed based on the customized strategy, the corresponding recycle bin folder is generated according to the current date, and the newly generated recycle bin folder is moved to the recycle bin directory. The backup is inserted into a mongo db, which is a database based on distributed file storage, and an operational record is maintained.
The RPC is used for calling each platform storage server to execute cleaning work of the recycle bin for the backup, a cleaning list in the backup to be cleaned in the recycle bin is deleted, a corresponding recycle bin catalog is deleted, and the recycle bin cleaning is performed according to preset time, wherein seven days are a preset value, which can be 7 days, and can also be set according to the requirement of a user.
It can be understood that the invention obtains the storage data of each storage area through the cleaning timer based on the cleaning instruction of the user, analyzes and generates the execution parameters corresponding to each storage area, wherein the invention divides the storage area into different storage areas, and each storage area is provided with an independent storage server, thereby realizing the division and deployment of different storage areas of the storage data, completing the distributed storage management and laying a cushion for improving the utilization rate of the storage space. Further, when the user issues the cleaning instruction, firstly, judging according to the acquired file type and file expiration time which are analyzed by the cleaning timer and the execution parameters of the storage path, determining one or more target storage areas to be cleaned, then, scanning the storage areas by combining with a customized strategy, namely, a file cleaning period, a file type and a storage path which are set by the user, screening out storage data which meet the execution parameters and the cleaning instruction, generating a scanning result, and cleaning the scanning result to complete the whole automatic cleaning process.
Therefore, a scanning result is generated by accessing the storage server of the target storage area through the cleaning timer based on the cleaning instruction and the execution parameters of the user, and cleaning work is carried out on the scanning structure.
It is to be understood that the same or similar parts in the above embodiments may be referred to each other, and that in some embodiments, the same or similar parts in other embodiments may be referred to.
It should be noted that in the description of the present invention, the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. Furthermore, in the description of the present invention, unless otherwise indicated, the meaning of "plurality" means at least two.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and further implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.
It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.
Those of ordinary skill in the art will appreciate that all or a portion of the steps carried out in the method of the above-described embodiments may be implemented by a program to instruct related hardware, where the program may be stored in a computer readable storage medium, and where the program, when executed, includes one or a combination of the steps of the method embodiments.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing module, or each unit may exist alone physically, or two or more units may be integrated in one module. The integrated modules may be implemented in hardware or in software functional modules. The integrated modules may also be stored in a computer readable storage medium if implemented in the form of software functional modules and sold or used as a stand-alone product.
The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, or the like.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and not to be construed as limiting the invention, and that variations, modifications, alternatives and variations may be made to the above embodiments by one of ordinary skill in the art within the scope of the invention.

Claims (9)

1. A storage distributed cleaning method based on cloud rendering, comprising:
acquiring a clearing instruction of a user, wherein the clearing instruction comprises a customized instruction; the customization instructions comprise customization strategies of how the user cleans the stored data, wherein the customization strategies comprise file cleaning periods, file types and storage paths set by the user;
based on a cleaning instruction of a user, acquiring storage data of each storage area through a cleaning timer, and analyzing and generating execution parameters corresponding to each storage area; wherein each storage area is provided with an independent storage server; the execution parameters include the storage path, file type and file expiration time;
judging a target storage area to be cleaned through a cleaning timer based on the execution parameters and the cleaning instruction;
obtaining a scanning result obtained by a storage server of a target storage area, wherein the scanning result is obtained based on the clearing instruction and the execution parameter;
and cleaning the scanning result of the target storage area.
2. The method of claim 1, wherein the purge instruction further comprises a preset system default purge rule.
3. The method of claim 1, wherein the cleaning the scan of the target storage area comprises:
acquiring and analyzing the scanning result through a cleaning timer, and generating job information of a user, wherein the job information represents the specific completion time of the current user of the current job; scanning the scanning result based on the operation information and the cleaning instruction through a cleaning timer to generate a cleaning list, and caching the cleaning list;
acquiring the cleaning list of the target storage area, and backing up the cleaning list;
generating a folder of a corresponding recycle bin according to the current date from the cleaning list, wherein the folder is stored in a recycle bin directory;
and calling a storage server of the target storage area to clean up the backup, and deleting the recycle bin directory.
4. A method according to claim 3, further comprising:
and deleting the recycle bin catalogue according to a preset period.
5. The method according to claim 1, wherein the obtaining the scan result obtained by the storage server of the target storage area, the scan result obtained based on the purge instruction and the execution parameter, includes:
and if the scanning result is empty, acquiring the last scanning result.
6. The method according to claim 1, characterized in that it comprises:
and caching the scanning result.
7. The method of claim 1, wherein the step of determining the position of the substrate comprises,
the storage area comprises a complete type and a storage server;
each storage area is used for executing different rendering tasks;
the complete type representation at least comprises one of each type of stored data required for completing the rendering task;
and the storage areas are isolated from each other.
8. The method of claim 2, wherein after the obtaining the storage server of the target storage area and before the scan result based on the purge instruction and the execution parameter, further comprising:
acquiring storage data of a target storage area, judging whether the storage data is an active user, if so, inquiring the ID of the active user, and determining the expiration time of the user;
and if not, screening based on the clearing instruction and the execution parameters to obtain a scanning result.
9. Storage distributed cleaning device based on cloud rendering, characterized by comprising:
the system comprises an acquisition module, a control module and a control module, wherein the acquisition module is used for acquiring a clearing instruction of a user, and the clearing instruction comprises a customization instruction; the customization instructions comprise customization strategies of how the stored data are cleaned by a user, wherein the customization strategies comprise file cleaning periods, file types and storage paths which are set by the user;
the distributed storage and analysis module is used for acquiring the storage data of each storage area through a cleaning timer based on a cleaning instruction of a user, and analyzing and generating execution parameters corresponding to each storage area; wherein each storage area is provided with an independent storage server; the execution parameters include the storage path, file type and file expiration time;
the judging module is used for judging a target storage area to be cleaned through a cleaning timer based on the execution parameters and the cleaning instruction;
the scheduling module is used for acquiring a scanning result obtained by a storage server of the target storage area, wherein the scanning result is obtained based on the clearing instruction and the execution parameters;
and the cleaning module is used for cleaning the scanning result of the target storage area.
CN202310666126.1A 2023-06-06 2023-06-06 Storage distributed cleaning method and device based on cloud rendering Pending CN116756101A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310666126.1A CN116756101A (en) 2023-06-06 2023-06-06 Storage distributed cleaning method and device based on cloud rendering

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310666126.1A CN116756101A (en) 2023-06-06 2023-06-06 Storage distributed cleaning method and device based on cloud rendering

Publications (1)

Publication Number Publication Date
CN116756101A true CN116756101A (en) 2023-09-15

Family

ID=87954502

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310666126.1A Pending CN116756101A (en) 2023-06-06 2023-06-06 Storage distributed cleaning method and device based on cloud rendering

Country Status (1)

Country Link
CN (1) CN116756101A (en)

Similar Documents

Publication Publication Date Title
US8214406B2 (en) Backup of incremental metadata in block based backup systems
US8413134B2 (en) Method, system and computer program for installing software products based on package introspection
US8886610B2 (en) Backup simulation for backing up filesystems to a storage device
US7552115B2 (en) Method and system for efficient generation of storage reports
US20070078914A1 (en) Method, apparatus and program storage device for providing a centralized policy based preallocation in a distributed file system
US20060136510A1 (en) Method and system for tracking changes in a document
DE112011104356T5 (en) Promote software images based on streaming technology
US20040015524A1 (en) Method and apparatus for managing digitally-stored media files
CN104408201A (en) Junk file cleaning method and device
CN112463450A (en) Incremental backup management method, system, electronic equipment and storage medium
CN108073720B (en) Data quality management system and method applied to big data system
US7680983B2 (en) Method of restoring data by CDP utilizing file system information
CN109063040B (en) Client program data acquisition method and system
CN111158957A (en) Virtual machine file recovery method, system, device and storage medium
CN116756101A (en) Storage distributed cleaning method and device based on cloud rendering
CN110633250A (en) Resource management system and method
CN110247801A (en) A kind of monitoring system and method for pair of cluster host
CN116578570B (en) Method, system and equipment for automatically optimizing table data structure layout
US7996717B1 (en) Compression of temporal dimension in a temporal storage device
US7603376B1 (en) File and folder scanning method and apparatus
CN110457279A (en) Off-line data scan method, device, server and readable storage medium storing program for executing
JPH09114708A (en) Operation history collection controlling method of operator in human interface terminal and device
CN110119562B (en) Method, equipment and system for managing model component files
CN113220592A (en) Processing method and device for automated testing resources, server and storage medium
JP3980748B2 (en) Network system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination