CN115905159A

CN115905159A - Business data file storage method, device and system based on storage strategy

Info

Publication number: CN115905159A
Application number: CN202211575406.3A
Authority: CN
Inventors: 陈鑫; 陈文建; 熊学武; 韦利东
Original assignee: Chengdu Pan Micro Network Technology Co ltd
Current assignee: Chengdu Pan Micro Network Technology Co ltd
Priority date: 2022-12-08
Filing date: 2022-12-08
Publication date: 2023-04-04

Abstract

The invention discloses a method, a device and a system for storing a business data file based on a storage strategy, and relates to the technical field of document distributed storage. The method comprises the steps of firstly receiving storage strategies which are from a file service front end and are set by a user for different business modules in a business system and business data physical files from a certain business module in the business system, and then carrying out corresponding storage processing on the business data physical files according to a local storage directory, a file encryption mode, a file storage mode and a local storage file backup mode in the storage strategies corresponding to the certain business module, so that the user only needs to pay attention to how the data fall to the ground, the enterprise realizes the storage cost is reduced, the files are more conveniently managed for the user, the files are classified and subjected to disaster recovery backup, the enterprise is further helped to better manage own file data, the file data is managed at low cost, and the method is convenient for practical application and popularization.

Description

Business data file storage method, device and system based on storage strategy

Technical Field

The invention belongs to the technical field of document distributed storage, and particularly relates to a method, a device and a system for storing a business data file based on a storage strategy.

Background

In the development of the traditional software industry, data of an enterprise needs centralized management and storage, and although the current cloud storage solution in the industry can solve the requirement, more personalized requirements of the enterprise cannot be met, wherein the requirements include the requirements of managing files in all business systems, classifying and storing the files according to certain rules, backing up cold and hot data regularly, performing dynamic storage capacity expansion when the capacity of a disk reaches certain limits, and preventing the data from being unavailable due to disk damage, and the like, namely the prior art specifically has the following problems: (1) Cloud storage only provides a general solution, cannot be well combined with industries, and cannot realize a multi-element storage scheme, such as automatically filing at a certain storage drive position according to a certain type of flow and a certain rule; (2) In the face of enterprises in different geographic positions, the access and storage performances are different, and the time consumption is long for a remote mechanism to access the files of the core mechanism, and the downloading is slow; (3) Physical file data in an enterprise are rarely stored in the cloud, and the risk of leakage of file contents may exist; (4) The disk is easy to damage in practical application, and the damaged disk data is unavailable, so that the service data cannot be accessed and downloaded, and meanwhile, when the disk is full, a dynamic capacity expansion mechanism cannot be realized; (5) The security of the file in the storage disk cannot be guaranteed, and the cost of butt joint encryption of enterprises is quite high; (6) Data cannot be well utilized in an enterprise, and data with the above T cannot be utilized to realize certain value for the enterprise.

Therefore, how to help an enterprise manage its own document data better is the same as how to manage its own document data at low cost, and a problem to be researched is urgently needed by those skilled in the art.

Disclosure of Invention

The invention aims to provide a business data file storage method, a business data file storage device and a business data file storage system based on a storage strategy, which are used for solving the problem that the existing cloud storage solution cannot meet the personalized requirements of enterprise users.

In order to achieve the purpose, the invention adopts the following technical scheme:

in a first aspect, a method for storing a service data file based on a storage policy is provided, which is executed by a file service backend, and includes:

receiving a storage strategy which is from a file service front end and is set by a user for different service modules in a service system, wherein the storage strategy comprises a local storage directory, a file encryption mode, a file storage mode and a local storage file backup mode;

receiving a service data physical file from a certain service module in the service system;

and according to a storage strategy corresponding to the certain service module, performing the following storage processing on the service data physical file: the method comprises the steps of locally storing the service data physical file by using a local storage directory in the storage strategy, encrypting the service data physical file by using a file encryption mode in the storage strategy, non-locally storing the service data physical file by using a file storage mode in the storage strategy, and backing up the locally stored service data physical file by using a local storage file backup mode in the storage strategy.

Based on the invention, a new storage scheme capable of meeting the personalized requirements of enterprise users is provided, namely, a storage strategy which is from the front end of a file service and is set by a user for different business modules in a business system and a business data physical file from a certain business module in the business system are received, and then the business data physical file is correspondingly stored and processed according to a local storage directory, a file encryption mode, a file storage mode and a local storage file backup mode in the storage strategy corresponding to the certain business module, so that the user only needs to pay attention to how the data falls on the ground, the storage cost of the enterprise is reduced, the files are more conveniently managed, classified and disaster-tolerant backed up, and the enterprise is further helped to better manage the file data of the enterprise, and the enterprise is convenient to actually apply and popularize like managing the file data of the enterprise at low cost.

In one possible design, receiving a service data physical file from a service module in the service system includes:

and receiving a first service data physical file uploaded by the front end of a certain service module in the service system through a hypertext transfer protocol (HTTP).

receiving a second service data physical file uploaded by a rear end of a certain service module in the service system through an encapsulation processing end, wherein the encapsulation processing end is used for uploading the second service data physical file as follows: the method comprises the steps of firstly obtaining a storage strategy corresponding to the rear end of a certain service module from the local, then uploading a second service data physical file from the rear end of the certain service module in an SDK mode by calling a software development kit of a storage package according to the storage strategy, and finally calling a local Remote Procedure Call (RPC) interface to interact service information of the second service data file.

In a possible design, the storing the service data physical file according to the storage policy corresponding to the certain service module includes:

and uniformly exposing an Application Programming Interface (API) of the storage packet through a Windows azure storage service StorageService, uniformly transmitting a storage strategy corresponding to a certain service module through a storage instance factory, and returning a corresponding storage instance so as to store and process the service data physical file based on the storage instance.

In a possible design, the storage policy further includes a DNS domain name adding rule and a CDN resource accelerated access mechanism, so that after the storage processing is performed on the service data physical file according to the storage policy corresponding to the certain service module, a user can select data of a nearby node for download access, and policy access restriction is performed on each stored bucket.

In a possible design, the storage policy further includes an analysis rule, so that after the storage processing is performed on the business data physical file according to the storage policy corresponding to the certain business module, the stored data can be periodically scanned, and the scanned content is detected and analyzed based on the analysis rule, so as to form a data report that can periodically provide an analysis result and a business prediction for a user.

In one possible design, the storage policy further includes a deployment solution supporting privatized storage or a client intranet deployment mechanism supporting business development by combining an open source storage mode MinIO and/or Ceph, so that a user can select a single-host single-hard-disk mode, a single-host multi-hard-disk mode or a multi-host multi-hard-disk mode to perform federal mode deployment, and remote dual-active and high-availability are supported.

In one possible design, the file encryption mode includes a mode supporting SM4/AES level encryption, a mode supporting amazon S3 standard encryption, a mode supporting third party manufacturer encryptor, and a mode supporting upload, download, encryption, and decryption, where upload, download, encryption, download encryption, upload and download decryption are upload encryption and download encryption;

the file storage mode comprises a mode of mounting a shared disk through an NAS and transmitting a file by using a zero copy of NIO when local object storage is selected, a mode of accessing an Tencent cloud object storage SDK when an Tencent cloud storage COS is selected, a mode of accessing an Aliskian object storage SDK when an Aliskian storage OSS is selected, a mode of accessing an Amazon service standard when an Amazon S3 standard is selected and a mode of accessing distributed storage service when distributed object storage is selected;

the local storage file backup mode comprises a mode of adopting a storage recovery algorithm RS erasure code and checking and checksum.

In a second aspect, a storage policy-based business data file storage device is provided, which is suitable for being arranged in a file service backend and comprises a first receiving module, a second receiving module and a storage processing module;

the first receiving module is used for receiving storage strategies which are from the front end of the file service and set by a user for different service modules in the service system, wherein the storage strategies comprise a local storage directory, a file encryption mode, a file storage mode and a local storage file backup mode;

the second receiving module is used for receiving a service data physical file from a certain service module in the service system;

the storage processing module is respectively in communication connection with the first receiving module and the second receiving module, and is configured to perform the following storage processing on the service data physical file according to a storage policy corresponding to the certain service module: the method comprises the steps of adopting a local storage directory in the storage strategy to locally store the service data physical file, adopting a file encryption mode in the storage strategy to encrypt the service data physical file, adopting a file storage mode in the storage strategy to nonlocal store the service data physical file, and adopting a local storage file backup mode in the storage strategy to backup the locally stored service data physical file.

In a third aspect, the present invention provides a storage policy-based service data file storage system, which includes a file service front end, a file service back end, and at least one service module of a service system;

the file service front end is used for transmitting the acquired storage strategies which are set by the user for different service modules in the service system to the file service back end;

the business module is used for transmitting the generated physical file of the business data to the file service back end;

the file service backend is respectively in communication connection with the file service front end and the at least one service module, and is configured to execute the service data file storage method according to any possible design in the first aspect or the first aspect.

The beneficial effect of above-mentioned scheme:

(1) The invention creatively provides a new storage scheme capable of meeting the personalized requirements of enterprise users, namely, a storage strategy which is from the front end of a file service and is set by a user for different business modules in a business system and a business data physical file from a certain business module in the business system are received, and then the business data physical file is correspondingly stored and processed according to a local storage directory, a file encryption mode, a file storage mode and a local storage file backup mode in the storage strategy corresponding to the certain business module, so that the user only needs to pay attention to how the data falls to the ground, the storage cost of the enterprise is reduced, the files are more conveniently managed, classified and disaster-tolerant backed up, and the enterprise is helped to better manage the file data of the enterprise just like managing the file data of the enterprise at low cost;

(2) The closed loop can be formed by diversified storage, backup, synchronization and encryption through a regularized thinking, users are guided to complete own complex requirements, and secondary development and expansion of customized requirements can be supported;

(3) The user can manage the own storage system by himself, the operation is simple and easy to operate, and the visual interface is supported to switch instances and dynamically configure capacity expansion nodes;

(4) High availability can be realized, the problem of slow data node access is solved, and the method is convenient for practical application and popularization.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a schematic flow chart of a method for storing a service data file based on a storage policy according to an embodiment of the present application.

Fig. 2 is a schematic diagram of information interaction and processing flows between a file service front-end and a service module back-end provided in an embodiment of the present application.

Fig. 3 is a schematic diagram of multi-node cross-region storage deployment according to an embodiment of the present application.

Fig. 4 is a schematic structural diagram of a service data file storage device based on a storage policy according to an embodiment of the present application.

Fig. 5 is a schematic structural diagram of a business data file storage system based on a storage policy according to an embodiment of the present application.

Detailed Description

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be briefly described below with reference to the accompanying drawings and the embodiments or the description in the prior art, it is obvious that the following description of the structure of the drawings is only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts. It should be noted that the description of the embodiments is provided to help understanding of the present invention, but the present invention is not limited thereto.

It will be understood that, although the terms first, second, etc. may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another. For example, a first object may be referred to as a second object, and a second object may similarly be referred to as a first object, without departing from the scope of example embodiments of the invention.

It should be understood that, for the term "and/or" as may appear herein, it is merely an associative relationship that describes an associated object, meaning that three relationships may exist, e.g., a and/or B may mean: a exists alone, B exists alone or A and B exist at the same time; also for example, a, B, and/or C, may indicate the presence of any one or any combination of a, B, and C; for the term "/and" as may appear herein, which describes another associative object relationship, it means that two relationships may exist, e.g., a/and B, may mean: a exists singly or A and B exist simultaneously; in addition, for the character "/" that may appear herein, it generally means that the former and latter associated objects are in an "or" relationship.

Example (b):

as shown in fig. 1 and 2, the service data file storage method based on the storage policy provided in the first aspect of the present embodiment may be, but is not limited to be, implemented by a Computer device having certain computing resources and serving as a backend of a file, for example, implemented by an electronic device such as a platform server, a Personal Computer (PC, which refers to a multipurpose Computer with a size, price and performance suitable for Personal use, a desktop Computer, a notebook Computer, a small notebook Computer, a tablet Computer, an ultrabook, and the like all belong to a Personal Computer), a smart phone, a Personal Digital Assistant (PDA), or a wearable device. As shown in fig. 1 and 2, the method for storing the service data file may include, but is not limited to, the following steps S1 to S3.

S1, receiving storage strategies which are from a file service front end and set by a user for different service modules in a service system, wherein the storage strategies comprise but are not limited to a local storage directory, a file encryption mode, a file storage mode, a local storage file backup mode and the like.

In the step S1, the file service front end is configured to obtain, in a human-computer interaction manner, storage policies set by a user for different service modules in a service system, and may specifically but not limited to be a foreground user terminal interface based on a BS architecture (generally, refer to a B/S structure, that is, a Browser/Server mode, which is a network structure mode after WEB is emerging, and a WEB Browser is a most main application software of a client, and this mode unifies the client, and centralizes a core part of system function implementation on a Server, thereby simplifying development, maintenance, and use of the system), for example, specifically, a WEB page or an APP (abbreviated as APP, which refers to an application) interface, and therefore, the storage policies input by the user may be obtained in a conventional human-computer interaction manner. In addition, the file serving front end may communicate the storage policy locally through conventional communication means.

The local storage directory is a local homing position set for a business data physical file of a certain business module in the business system, such as "/opt/xx", so that the business data physical file is filed under the locally stored "/opt/xx" in a naming mode of "name + time".

The file Encryption mode includes, but is not limited to, a mode supporting SM4 (which is the first commercial cryptographic algorithm in China, and is also a symmetric Encryption algorithm)/AES (Advanced Encryption Standard, also called Rijndael Encryption method, and is a block Encryption Standard adopted by the federal government of the united states) level Encryption, a mode supporting amazon S3 (Simple Storage Service, a set of Storage Service standards provided by amazon, S3 can provide Storage services for different users and different scenes, such as data lakes, websites, mobile applications, backup recovery of general data, big data analysis, and the like, and each big Storage and cloud manufacturer needs to extend the Storage on this Standard), a mode supporting an encryptor of a third party manufacturer, a mode supporting upload, download, encryption, and decryption, and the like, wherein the upload, download, decryption, upload, and decryption refer to upload Encryption, download Encryption, upload, download, and download Encryption, and the like. In this way, the storage scheme provided by this embodiment can support SM4/AES level encryption, amazon S3 standard encryption such as a KMS (Key managmentary service) encryption machine, third party vendor encryption machines (for example, greenshield and Tianyi, etc.), fast access for clients without additional development, and upload, download, encryption and decryption requirements through visual management configuration. For example, to implement decryption and download encryption of uploading, after the rule is selected, only the rule needs to be used by the background, and different encryption examples are stored in a policy design mode, so that different encryption and decryption rules (such as uploading, decryption, downloading encryption and decryption, not performing encryption and decryption, uploading, encryption and downloading, decryption and the like) are supported for different business files, for example, file data of the A document flow is encrypted by AES, and uploading, encryption and downloading, decryption are performed; meanwhile, other service data are encrypted by Yisaitong, and support can be provided in parallel.

The file Storage method includes, but is not limited to, a method of mounting a shared disk by a NAS (Network Attached Storage, literally simply referred to as a device connected to the Network and having a data Storage function, and therefore also referred to as a "Network Storage", which is a dedicated data Storage server) and using a zero copy of NIO (Java non-blocking IO, which is a New API provided by JDK, from JDK1.4, java provides a series of New features of improved input/output, which is collectively referred to as NIO, that is, new IO, and is synchronously non-blocking) for file transmission, a method of accessing a Cloud Object Storage SDK (Software Development Kit) when selecting COS (Cloud Object Storage), a method of accessing a secure Cloud Object Storage Service, a secure Storage Service, a low-cost secure Storage Service, a method of accessing a Software Development Kit (Software Development Kit), a method of selecting an aristo Storage OSS (Object Storage Service, 99999999999999999999993% of mass data, and a method of accessing a distributed Storage Service when selecting a local Object Storage COS (Cloud Object Storage Service, 999999993% of distributed Storage Service, and so on-a distributed Storage map Service. Therefore, through the combination of integration of different third parties (such as cloud manufacturers like Ariiyun, tencent cloud, huashi cloud and Amazon S3) and self-developed distributed storage and rules, enterprises are helped to land on the ground of data storage from different business modules, storage schemes can be selected by users, and pluggable realization of storage media is achieved through strategy design and adapter modes.

The local storage file backup mode includes, but is not limited to, a mode of using a storage recovery algorithm RS erasure code and checking and checksum. The RS (Reed-Solomon) Erasure codes are the existing Erasure codes (Erasure Code, EC, which is a forward error correction technology and is mainly applied to network transmission to avoid packet loss, a storage system utilizes the Erasure codes to improve storage reliability, compared with multi-copy replication, the Erasure codes can obtain higher data reliability with smaller data redundancy, but the coding mode is more complex and needs a large amount of calculation, the Erasure codes can only tolerate data loss and cannot tolerate data tampering, and the Erasure codes are named as the Erasure codes, the Erasure Code is a coding technology which can restore n parts of original data into the original data by increasing m parts of data and by any n parts of data in n + m parts, namely if any part of data less than or equal to m parts of data fails, the remaining data can still be restored. From a data function perspective, the protection provided by erasure codes can be expressed by the following simple formula: n = k + m, where the variable "k" represents the value of the original data or symbol, the variable "m" represents the value of the extra or redundant symbol added after the failure to provide protection, and the variable "n" represents the total value of the symbols created after the erasure coding process. For example, there are 16 disk drives, that is, n, there are 10 original files, and in a modular manner, k,16=10+ m, where m is the number of recoverable check blocks, so m is 6, any 6 files are unavailable, the original files can be recovered, in an extreme case, 6 original files are damaged by 10 files, and 6 original files that are damaged by adding 6 check blocks to 4 original files can be recovered. Therefore, by adopting a storage recovery algorithm RS erasure code and a mode of checking and checking, data can be protected from hardware failure and silent data damage, and even if a disk with half blocks is lost, the data can be recovered to ensure that the data can be accessed.

In addition, the file service front end can realize expansion configuration centralization through ETCD so as to conveniently carry out multi-node storage expansion during the disk early warning period. The aforementioned ETCD (which is an open-source project initiated by the CoreOS team in 2013, 6, with the goal of building a highly available distributed key-value database) is an open-source, distributed key-value-pair data storage system, provides for registration and discovery of shared configurations, services, and has the following features: (a) full replication: each node in the cluster may use a complete archive; (b) high availability: the ETCD can be used for avoiding single-point failure or network problems of hardware; (c) consistency: each read returns the latest write across multiple hosts; (d) simple: including a well-defined, user-oriented API (Application Programming Interface); (e) safety: an automated TLS (Transport Layer Security protocol) with optional client certificate authentication is implemented; (f) fast: a reference speed of 10000 writes per second; (g) reliable: the method uses the Raft algorithm (namely, a distributed algorithm which is simpler, more convenient and easier to understand and mainly solves the consistency problem in the distribution) to realize the strong consistency and high availability of the service storage directory.

And S2, receiving a service data physical file from a certain service module in the service system.

In step S2, the service module is considered to have a front end and a back end, so that specifically, the receiving of the service data physical file from a certain service module in the service system includes, but is not limited to: receiving a first service data physical file uploaded by a front end of a certain service module in the service system through a hypertext Transfer Protocol (HTTP), which is a simple request-response Protocol; and receiving a second service data physical file uploaded by a rear end of a certain service module in the service system through an encapsulation processing end, wherein the encapsulation processing end is used for uploading the second service data physical file as follows: the method comprises the steps of firstly obtaining a storage strategy corresponding to the rear end of a certain service module from the local, then uploading a second service data physical file from the rear end of the certain service module in an SDK (software development kit) mode by calling a storage package according to the storage strategy, and finally calling a local Remote Procedure Call (RPC) interface to interact service information of the second service data file.

S3, performing the following storage processing on the service data physical file according to the storage strategy corresponding to the certain service module: the method comprises the steps of adopting a local storage directory in the storage strategy to locally store the service data physical file, adopting a file encryption mode in the storage strategy to encrypt the service data physical file, adopting a file storage mode in the storage strategy to nonlocal store the service data physical file, and adopting a local storage file backup mode in the storage strategy to backup the locally stored service data physical file.

In step S3, specifically, the storing the service data physical file according to the storage policy corresponding to the certain service module includes: and uniformly exposing an Application Programming Interface (API) of the storage packet through a Windows azure storage service StorageService, uniformly transmitting a storage strategy corresponding to a certain service module through a storage instance factory, and returning a corresponding storage instance so as to store and process the service data physical file based on the storage instance. Through the steps S1 to S3, the user can set the storage instance supported by the system by himself, for example, the combination of local storage and aricloud storage, set the rule for storing the xx-class service data, and provide a management interface to support one-key switching of the storage instance, one-key synchronization data, and one-key timing backup. As rule 1: all attachments in a certain type of flow are filed under the locally stored "/opt/xx" according to the name + time; generating a backup rule, and periodically backing up data under a source "/opt/xx" to a source "xxx"; data access rules, access to full data or half a year/n month ago data, and so on.

Therefore, based on the business data file storage method described in the foregoing steps S1 to S3 and based on the storage policy, a new storage scheme is provided that can meet the personalized requirements of enterprise users, that is, a storage policy from the front end of a file service and set by a user for different business modules in a business system and a business data physical file from a certain business module in the business system are received first, and then the business data physical file is subjected to corresponding storage processing according to a local storage directory, a file encryption manner, a file storage manner and a local storage file backup manner in the storage policy corresponding to the certain business module, so that the user only needs to pay attention to how to fall on the ground of data, the enterprise realizes storage cost is reduced, files are managed for the user more conveniently, and the files are classified and disaster-tolerant backed up, thereby helping the enterprise manage its own file data better, just like managing its own file data at low cost, and facilitating practical application and popularization. In addition, a closed loop can be formed through diversified storage, backup, synchronization and encryption by a regularized thinking, a user is guided to complete own complex requirements, and secondary development and expansion of customized requirements can be supported; and the user can manage the own storage system by himself, the operation is simple and easy to operate, a visual interface is supported to switch instances, and capacity expansion nodes are dynamically configured.

In a possible design, the storage policy further includes a DNS (domain name system, which is a service of the internet, and serves as a distributed database that maps domain names and IP addresses to each other, so that people can access the internet more conveniently), a domain name plus setting rule and a CDN (Content Delivery Network, that is, a Content Delivery Network, and a basic idea thereof is to avoid bottlenecks and links on the internet that may affect data transmission speed and stability as much as possible, so that Content transmission is faster and more stable) resource accelerated access mechanism, so that after the storage processing is performed on the service data physical file according to the storage policy corresponding to the certain service module, a user can select data of a nearby node for download access, and policy access restriction is performed on each stored bucket. The policy access restrictions include, but are not limited to, read-only, read-write, and admin settings. The following scenario is assumed to exist: if the distance of the mechanism B is long and the speed of accessing the mechanism A data is increased, the node of the mechanism A is used as a certain node of the CDN to accelerate by adding a rule and a CDN resource accelerating access mechanism to the DNS domain name, and a piece of temporary data is cached at a nearby node; as shown in fig. 3, the shanghai institution is a headquarter, the beijing institution is a branch, and requires that the beijing users upload and download data without being affected, but only access data of the beijing nodes, and the shanghai users can access data of any region, so that the shanghai nodes can be used as master nodes to read data, the beijing nodes are used as slave nodes to write data, and the beijing nodes can cache a piece of hot spot temporary data and control access rights of the beijing buckets and the shanghai buckets.

In a possible design, the storage policy further includes an analysis rule, so that after the storage processing is performed on the business data physical file according to the storage policy corresponding to the certain business module, the stored data can be periodically scanned, and the scanned content is detected and analyzed based on the analysis rule, so as to form a data report that can periodically provide an analysis result and a business prediction for a user. Therefore, by scanning the content of certain types of files, extracting keywords, then extracting the keywords, completing filtering data by sensitive detection, obtaining a series of sampled samples, then marking keyword labels, performing retrieval, intelligent recommendation and the like, and finally predicting the development trend of companies through text content.

In one possible design, the storage policy further includes a deployment solution (such as NAS) supporting privatized storage or a deployment solution supporting federated active and high availability in combination with an open source storage mode MinIO (which is an object storage server published under Apache License v2.0, which is compatible with Amazon S3 cloud storage service, which is most suitable for storing unstructured data such as photos, videos, log files, backup and container/VM images, etc., the size of the object may be from several KB to maximum 5 TB) and/or Ceph (which is an impressive file system candidate added to Linux), which is a distributed file system capable of adding copy and fault tolerance functions while maintaining POSIX compatibility) as a client intranet deployment mechanism that is open to business, so that a user can select a single-host single-hard disk mode, a single-host multi-hard disk mode or a multi-host multi-hard disk mode for deployment. For example, a multi-hard disk multi-host mode is adopted, the number of drives is 4-16 as the total number of nodes, and data is stored and accessed by different nodes.

As shown in fig. 4, a second aspect of this embodiment provides a virtual device for implementing the method for storing a service data file according to any possible design in the first aspect or the first aspect, which is suitable for being disposed in a file service backend and includes a first receiving module, a second receiving module, and a storage processing module;

the first receiving module is used for receiving storage strategies which are from a file service front end and set by a user for different service modules in a service system, wherein the storage strategies comprise a local storage directory, a file encryption mode, a file storage mode and a local storage file backup mode;

the storage processing module is respectively in communication connection with the first receiving module and the second receiving module, and is configured to perform the following storage processing on the service data physical file according to a storage policy corresponding to the certain service module: the method comprises the steps of locally storing the service data physical file by using a local storage directory in the storage strategy, encrypting the service data physical file by using a file encryption mode in the storage strategy, non-locally storing the service data physical file by using a file storage mode in the storage strategy, and backing up the locally stored service data physical file by using a local storage file backup mode in the storage strategy.

For the working process, working details and technical effects of the foregoing apparatus provided in the second aspect of this embodiment, reference may be made to the first aspect or any possible design of the method for storing the service data file in the first aspect, which is not described herein again.

As shown in fig. 5, a third aspect of this embodiment provides an entity system applying the first aspect or any possible design of the method for storing a service data file in the first aspect, where the entity system includes a file service front end, a file service back end, and at least one service module of a service system;

For the working process, the working details, and the technical effects of the foregoing system provided in the third aspect of this embodiment, reference may be made to the first aspect or any possibly designed service data file storage method in the first aspect, which is not described herein again.

Finally, it should be noted that: the above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A business data file storage method based on a storage strategy is characterized in that the method is executed by a file service backend and comprises the following steps:

and performing the following storage processing on the service data physical file according to a storage strategy corresponding to the certain service module: the method comprises the steps of adopting a local storage directory in the storage strategy to locally store the service data physical file, adopting a file encryption mode in the storage strategy to encrypt the service data physical file, adopting a file storage mode in the storage strategy to nonlocal store the service data physical file, and adopting a local storage file backup mode in the storage strategy to backup the locally stored service data physical file.

2. The method of claim 1, wherein receiving a physical file of service data from a service module in the service system comprises:

3. The method for storing service data files according to claim 1, wherein receiving a service data physical file from a service module in the service system comprises:

4. The method for storing the service data file according to claim 3, wherein the storing the service data physical file according to the storage policy corresponding to the certain service module comprises:

and uniformly exposing an Application Programming Interface (API) of the storage packet through a Windows Azure storage service StorageService, uniformly transmitting a storage strategy corresponding to a certain service module through a storage instance factory StorageFactory, and returning a corresponding storage instance so as to store and process the service data physical file based on the storage instance.

5. The method according to claim 1, wherein the storage policy further includes a DNS domain name adding rule and a CDN resource accelerated access mechanism, so that after the storage processing is performed on the service data physical file according to the storage policy corresponding to the certain service module, a user can select data of a nearby node for download access and make policy access restrictions on each stored bucket.

6. The method according to claim 1, wherein the storage policy further includes an analysis rule, so that after the storage processing is performed on the physical file of the business data according to the storage policy corresponding to the certain business module, the stored data can be scanned periodically, and the scanned content is detected and analyzed based on the analysis rule, thereby forming a data report that can provide the analysis result and the business prediction for the user periodically.

7. The method for storing the service data file according to claim 1, wherein the storage policy further includes a deployment solution supporting private storage or a client intranet deployment mechanism supporting service second development by combining an open source storage mode MinIO and/or Ceph, so that a user can select a single-host single-hard-disk mode, a single-host multi-hard-disk mode or a multi-host multi-hard-disk mode for federal mode deployment, and support remote dual-active and high-availability.

8. The business data file storage method according to claim 1, wherein the file encryption modes include a mode supporting SM4/AES level encryption, a mode supporting amazon S3 standard encryption, a mode supporting third party manufacturer encryptor, and a mode supporting upload and download encryption and decryption, wherein the upload and download encryption and decryption means upload encryption and download decryption, upload decryption and download encryption and upload and download encryption;

the file storage mode comprises a mode of mounting a shared disk through an NAS and transmitting a file by using a zero copy of NIO when local object storage is selected, a mode of accessing an Tencent cloud object storage SDK when an Tencent cloud storage COS is selected, a mode of accessing an Aliskive cloud object storage SDK when an Alice cloud storage OSS is selected, a mode of accessing an Amazon service standard when an Amazon S3 standard is selected and a mode of accessing a distributed storage service when distributed object storage is selected;

9. A business data file storage device based on a storage strategy is characterized by being suitable for being arranged in a file service back end and comprising a first receiving module, a second receiving module and a storage processing module;

10. A business data file storage system based on storage strategy is characterized by comprising a file service front end, a file service back end and at least one business module of a business system;

the file service front end is used for transmitting the acquired storage strategies which are set for different service modules in the service system by a user to the file service back end;

the business module is used for transmitting the generated business data physical file to the file service back end;

the file service back end is respectively in communication connection with the file service front end and the at least one service module, and is configured to execute the service data file storage method according to any one of claims 1 to 8.