CN111736775B - Multi-source storage method, device, computer system and storage medium - Google Patents
Multi-source storage method, device, computer system and storage medium Download PDFInfo
- Publication number
- CN111736775B CN111736775B CN202010575391.5A CN202010575391A CN111736775B CN 111736775 B CN111736775 B CN 111736775B CN 202010575391 A CN202010575391 A CN 202010575391A CN 111736775 B CN111736775 B CN 111736775B
- Authority
- CN
- China
- Prior art keywords
- service
- storage
- data
- information
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
Abstract
The invention relates to artificial intelligence, and provides a multi-source storage method, a device, a computer system and a storage medium, wherein the method comprises the following steps: the method comprises the steps of firstly receiving service information which comprises types and service data and is sent by a client, then determining service storage modes according to the types of the service information, wherein each type of the service information corresponds to one service storage mode, and storing the service data according to the determined service storage mode. The invention distinguishes the data source by receiving the service information sent by the client and the equipment information of the client, then determines the service storage mode according to the type of the service information to determine the service storage mode of the storage service, hides the technical complexity of accessing various storage services, realizes the independence with the specific storage service, and the service system can store the service data after selecting according to the use scene, thereby bringing convenience for the access of files and file streams. The invention also relates to a blockchain technique, and the service data can be stored in blockchain nodes.
Description
Technical Field
The invention relates to the technical field of artificial intelligence data processing, in particular to a multi-source storage method, a multi-source storage device, a computer system and a storage medium.
Background
In the big data era, the data storage requirement is higher and higher, wherein object storage is a distributed storage service for storing mass data, and the object storage is widely applied to various industries and has the characteristics of safety, low cost, high reliability and the like. However, the distributed storage service is usually required to serve a plurality of objects, and the service objects often have different requirements on the storage file or the storage environment, which requires the improvement of the capability of the object storage service to adapt to different service objects. When a company business system interfaces to different systems in various places, different technical requirements or use specifications are usually stored in files in various places, or a storage system of a specific network environment needs to be used because of field network environment limitations. This results in the development team accessing different storage service technologies in different environments, which results in repetitive work. Therefore, how to better support the storage service of multiple storage sources is a problem to be solved by those skilled in the art.
Disclosure of Invention
The present invention is directed to a multi-source storage method, apparatus, computer system and storage medium, which solve the above problems of the prior art, and can be applied to the field of smart cities, thereby promoting the construction of smart cities.
In order to achieve the above object, the present invention provides a multi-source storage method, comprising the steps of:
s10, receiving service information sent by a client, wherein the service information comprises types and service data;
s20, identifying the type of the service information, and determining a service storage mode according to the type of the service information, wherein each type of the service information corresponds to one service storage mode;
and S30, storing the service data according to the determined service storage mode.
Further, the step S10 further includes receiving device information of the client sent by the client, matching the device information of the client with a preset list, and if the device information of the client is matched with the preset list, receiving the service information sent by the client by using a unified interface.
Further, the preset list includes preset client information;
the matching the device information of the client with a preset list comprises:
matching the equipment information of the client with preset client information in the preset list, and if the equipment information of the client is matched with the preset client information in the preset list, receiving the service information sent by the client by adopting a unified interface;
the preset list is stored in a block chain.
Further, the type of the service information includes API call and SDK call; the API calls correspond to a first service storage mode, and the SDK calls correspond to a second service storage mode.
Further, the step S30 stores the service data according to the determined service storage mode, including: when the type of the service information is API call, a first service storage mode is adopted to store the service data: decoding the service data to obtain a file name, a file identifier, file data and a type cluster, converting the file name, the file unique identifier, the file data and the type cluster into a file stream, and storing the file stream into a database corresponding to the type cluster, wherein each type cluster corresponds to a corresponding database.
Further, the step S30 stores the service data according to the determined service storage mode, including: when the type of the service information is called by the SDK, a second service storage mode is adopted to store the service data: decoding the service data, acquiring file data, a file identifier and a type cluster, inquiring a preset database according to the data volume of the file data, acquiring a database matched with the type cluster, acquiring an address of the database, generating a database identifier corresponding to the database according to the database address, sending the service data to the database according to the database identifier, recording the storage information of the service data, generating a storage log, and sending response information to the client.
In order to achieve the above object, the present invention provides a multi-source storage device, which includes a receiving unit, an identifying unit, and a processing unit, where the receiving unit is configured to receive service information sent by a client, where the service information includes a type and service data, the identifying unit is configured to identify the type of the service information, and determine a service storage mode according to the type of the service information, where each type of the service information corresponds to one service storage mode, and the processing unit is configured to store the service data according to the determined service storage mode.
Further, the type of the service information includes API call and SDK call; the API calls correspond to a first service storage mode, and the SDK calls correspond to a second service storage mode.
In order to achieve the above object, the present invention further provides a computer system comprising a plurality of computer devices, each computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processors of the plurality of computer devices collectively implementing the steps of the aforementioned method when executing the computer program.
In order to achieve the above object, the present invention also provides a computer-readable storage medium comprising a storage data area storing data created according to use of blockchain nodes and a storage program area storing a computer program, wherein the computer program stored in the storage medium implements the steps of the aforementioned method when executed by a processor.
By adopting the technical scheme, compared with the prior art, the invention has the following beneficial effects:
according to the multi-source storage method, the multi-source storage device, the computer system and the storage medium, the data source is distinguished by receiving the service information sent by the client and the equipment information of the client, then the service storage mode is determined according to the type of the service information so as to determine the service storage mode of the storage service, the technical complexity of accessing various storage services is hidden, the independence of the storage service is realized, the service system can store the service data after being selected according to the use scene, convenience is brought to the access of files and file streams, when the multi-source storage method, the multi-source storage device, the computer system and the storage medium are deployed and used in different areas, the service codes stored by the files do not need to be changed, and the development and production efficiency is improved.
Drawings
FIG. 1 is a flow chart of a multi-source storage method of the present invention;
FIG. 2 is a diagram illustrating the calling process of the multi-source storage method of the present invention;
FIG. 3 is a block diagram of an embodiment of a multi-source storage apparatus of the present invention;
fig. 4 is a hardware architecture diagram of one embodiment of the computer apparatus of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
The multi-source storage method, the device, the computer system and the storage medium provided by the invention distinguish the data source by receiving the business information sent by the client and the equipment information of the client, then determine the business storage mode according to the type of the business information to determine the business storage mode of the storage service, hide the technical complexity of accessing various storage services, realize the independence with the specific storage service, store the business data after the business system can be selected according to the use scene, bring convenience for the access of files and file streams, do not need to change the business codes stored by the files when being deployed and used in different areas, and improve the efficiency of development and production.
Example one
The invention provides a storage service supporting multiple storage sources, and referring to fig. 1, a multi-source storage method of the invention is shown, in step S10, business information sent by a client is received; the service information includes types and service data, the service information may be different service scenes in actual work, and may be account information, operation information, transaction information, or the like, the mobile terminal device and the terminal device may be distinguished from the device information of the client, and further, the number code of the device may be directly obtained, and the service data includes documents, audio and video, application programs, and the like.
In this embodiment, the type of the service information includes an API (Application Programming Interface) call and an SDK (Software Development Kit) call, where the API call performs a service through a REST (Representational State Transfer) Interface, the API call provides capabilities of an Application program and a developer to access a set of routines based on certain Software or hardware through a predefined function, and does not need to access a source code or understand details of an internal working mechanism, the SDK call performs a service through a Java package (Java ARchive) integration, and the SDK call includes a set of Development tools used when Application Software is established by a Software package, a Software framework, a hardware platform, an operating system, and the like.
For smaller files, the client can use an API calling mode to carry out service through an REST interface, such as a scene when files below 4MB are accessed; for larger files, the client can provide services in a mode of integrating jar packages through an SDK calling mode, and the method is suitable for accessing use scenes below 500 MB. The API calling mode can form an interface document with a standard format and content requirements, the back end receives legal data transmitted by the front end and returns data conforming to the standard according to an interface protocol, the front end transmits the data conforming to the standard according to the interface protocol and processes the data returned by the back end according to the display requirements, and the content of the interface document can comprise: interface description, request mode (GET/POST/RETRIEVE/UPDATE), interface address, request parameter (including parameter type and limitation condition), return parameter description (return packet structure), etc.
When a client needs to transmit a small file, such as an office file, a file book file, a common resolution picture and the like, the client can use an API calling mode, and the API calling mode can be suitable for accessing files below 4MB through an REST interface. When a client needs to transmit a large and small file, such as a video or a high-resolution picture, an SDK calling mode can be used, the SDK calling mode provides service through a mode of integrating jar, and the method is applicable to a use scene of the file up to 500 MB.
The client side can send service information, information and data to the application layer in an API calling mode, each layer in the network responds in sequence, and finally network data transmission is completed, the interface input parameters can include file names, file Unique identifiers, base64 coded file contents and other calling description information, the file Unique Identifier key can be a character string, the transmitted files can be generated according to UUIDs (Universal Unique Identifier), and data reading among application programs is realized by sending formatted files in the file transmission.
In this embodiment, the storage service is integrated, because each storage source has its own read interface, and a way of synchronizing data or accessing an opposite program is performed between two independent systems in an interface access manner, each interface can be docked to a unified interface to read data in the storage source, the storage source can be a resource pool of the storage service of the system, a series of storage services are added to the resource pool as data resources, the unified interface adapts the interfaces of the storage sources to read base data, the storage service is integrated, the unified interface can enter relevant information into the storage source according to a docking demand table, the demand information is stored in the docking demand table, interactive contents can be clarified through right control on the docking demand table, an insert operation or a select operation can be performed on the docking demand table, and data written into the docking demand table is an operation that a user needs to perform.
When the docking operation is carried out, relevant information can be input according to the docking requirement table, and the information can comprise: the method comprises the steps of uploading file types, file size ranges, storage service types of selected files and the like, further determining a docking service mode, wherein commonly used file types comprise text files, video files, audio files, picture files, system files and executable files, and the storage service types can be block storage, file storage or object storage. The two-dimensional table with a mathematical relationship is a collection of various attributes of homogeneous entities, each entity corresponds to a row in the table, called a tuple in the relationship, corresponding to a general record, and a column attribute in the table, called Field, corresponding to a data item in the general record, also called column, field.
For further control, step S10 further includes receiving device information of the client sent by the client, matching the device information of the client with a preset list, and if matching, receiving the service information sent by the client by using a unified interface, through the integrated gateway module, parsing all the invocation requests by using the gateway module, controlling according to the preset list such as a white list, filtering out illegal access requests, performing Encryption and decryption operations on access to sensitive Data, and supporting multiple Encryption and decryption modes such as Data Encryption Standard (DES), RSA, national password, and the like. In order to guarantee performance, access flow of each client can be controlled, different performance slopes are given according to weight values distributed by a system, and important clients are given higher priority. Meanwhile, heartbeat packages can be sent to the back-end storage service, availability scanning can be carried out, and if the conditions of unavailable service and the like occur, service early warning can be carried out in a mode of sending short messages and mails.
In this embodiment, the preset list includes preset client information; the matching the device information of the client with a preset list comprises: matching the equipment information of the client with preset client information in the preset list, and if the equipment information of the client is matched with the preset client information in the preset list, receiving the service information sent by the client by adopting a unified interface; the preset list is stored in a block chain.
The request routing forwarding can be performed according to the access URL (Uniform Resource Locator), and the stored file is guided to a specific storage service system according to different storage service requirements, so that the transmission of the stored file is realized. The URL application is based on an HTTP (Hypertext Transfer Protocol) or HTTPS (Hypertext Transfer Protocol over secure session Layer) mode to transmit data, a server side calls a client side according to a specified parameter name through exposing an address, a parameter name and a code, and the data is coded and encapsulated in the calling process.
The route forwarding can firstly carry out bitwise AND operation on the corresponding subnet mask in the local routing table, then query is carried out in the local routing table, if the subnet mask is the same as the local IP, the routing forwarding has already reached the destination end, and the data is analyzed by the current route; if the calculated IP address is not the local IP address, the IP is the next-hop routing IP, the routing forwarding is continued, and if the next-hop address cannot be inquired in the current router, the default next-hop IP is switched to. Routers may need to segment data packets to fit the frame size of the underlying network, and may drop data packets in the event of an overflow.
The function module of the service dispatching center is established for storing the service system, and the function module can complete the related functions of service forwarding, resource dispatching, file access, log recording and the like. And forwarding the request to access different storage modules according to the request path URL, wherein each storage module is realized by accessing a specific storage service and is served by integrating the client of the storage service or directly calling an interface provided by the client. And according to one or more storage modes required by the client, the scheduling center selects a storage service system with lower current resource load and better health degree to perform file access operation. For each operation, the service dispatching center records file information and operation logs, and can effectively track system operation records.
The storage service requirements may be classified by different storage methods, which may include a sequential storage method, a chained storage method, an index storage method, and a hash storage method. The sequential storage method stores logically adjacent nodes in physically adjacent storage units, and the logical relationship among the nodes is embodied by the adjacency relationship of the storage units; the link storage method does not require that logically adjacent nodes are also adjacent in physical location, and the logical relationship between the nodes is represented by an additional pointer field; the Index storage method is characterized in that an additional Index table is also established while node information is stored, the Index table is composed of a plurality of Index items, and if each node has one Index item in the Index table, the Index table is called as a Dense Index (Dense Index). If a group of nodes only corresponds to one Index entry in the Index table, the Index table is called a sparse Index (Spare Index); the hash storage method can directly calculate the storage address of the node according to the key word of the node, the hash data access speed is higher than that of an array, because the storage position of the data in the array can be found according to partial content of the stored data, the data access can be quickly realized, the ideal hash access speed is very quick, and unlike the traversal process in the array, partial elements of the content in the storage array are used as the input of a mapping function, the output of the mapping function is the position of the stored data, and the realization of traversing the array is omitted due to the access speed.
When multi-source storage is carried out, an API calling mode or an SDK calling mode can be selected according to an application scene by a client to be connected with the unified interface, and reading and writing of a storage file are completed, so that the storage system provides services through the unified interface, service codes do not need to be changed even if storage services are switched, and the use benefit of the storage services is improved.
In step S20, identifying the type of the service information, and determining a service storage mode according to the type of the service information;
each type of the service information corresponds to a service storage mode, so that different types of service information are stored according to the corresponding service storage modes.
In the present embodiment, the application scenarios classify the types of service information according to the size of the file. In this embodiment, the types of the service information include an API call and an SDK call, where the API call corresponds to the first service storage mode, the SDK call corresponds to the second service storage mode, so as to meet different call requirements of the client, an API call mode or an SDK call mode may be selected according to an application scenario for docking, and the API call mode is served through a REST (Representational State Transfer) interface.
And storing the service data according to the determined service storage mode in step S30.
When the type of the service information is API call, a first service storage mode is adopted to store the service data: decoding the service data, acquiring file names, file identifiers, file data and type clusters, converting the file names, the file unique identifiers, the file data and the type clusters into file streams, and storing the file streams into databases corresponding to the type clusters, wherein each type cluster corresponds to a corresponding database, and the type clusters comprise Ali cloud, huacheng cloud, peace cloud and the like.
The method can be used for decoding operation after receiving the file and converting the file into a file stream, the file stream can be divided into an input stream (reading the file) and an output stream (writing the file) according to functions, the file stream can be divided into a character stream (reading a character array) and a byte stream (reading a byte array) according to operation contents, the file can be read by using the byte stream, the file can be written by using the byte output stream, the file can be copied by using the byte input stream and the byte output stream, and the object serialization and the deserialization can be realized by using the object input output stream. Then storing the data into a corresponding storage system, taking an interface for uploading files as an example, the reference of the interface can be correspondingly referred to as the following table:
the interface message of the API calling mode may be sent in the form of json character string, and the format may refer to the following:
and returning a corresponding result in an interface calling mode, wherein the corresponding result can comprise a URL field of a file access address, and the following references are made:
the method is characterized in that an SDK calling mode is adopted when a large file is uploaded or a fragment uploading function needs to be supported, the method can be used by integrating a jar package calling related method, the jar package is a file format irrelevant to a platform, a plurality of files can be synthesized into one file, a plurality of Java applets and required components (class files, images and sounds) of the Java applets can be bound into the jar package, and then the Java applets and the required components (class files, images and sounds) can be downloaded into a browser as a single simple HTTP transaction, so that the downloading speed is greatly improved. The method parameters in the jar package comprise file objects, file unique identifiers, other description calling information and the like, and the objects returned by running the jar package comprise the returning information of method calling, including response codes, response information, file addresses, storage types and the like.
The interface calling code for uploading the file by the SDK calling mode can adopt the following steps:
StoreService service＝new StoreService()；
service. Setconfig (appId, appName, "001", "zhang");
File file＝new File("D:/test/aa.txt")；
StoreResponse response＝service.upload(file,sKey)；
the returned StoreResponse object can include return information of the method call, and the return information can include a response code, response information, a file address, a storage type and the like.
In step S30, storing the service data according to the determined service storage mode, when the type of the service information is SDK call, storing the service data by using a second service storage mode: decoding the service data, acquiring file data, file identification and type clusters, inquiring a preset database according to the data volume of the file data, acquiring the database matched with the type clusters, so as to obtain whether the type clusters such as Baidu cloud, ali cloud, safety cloud and the like have enough storage space and other state conditions, acquiring the addresses of the database, generating database identification corresponding to the database according to the database addresses, sending the service data to the database according to the database identification, recording the storage information of the service data, generating a storage log, and sending response information to the client.
Specifically, when the SDK calling method is adopted, reference is made to fig. 2 (the storage service takes cloud storage as an example) for a calling process between the SDK calling method and the storage system of the file, where the detailed steps include the following:
1. firstly, calling access parameters, directly calling an integrated SDK in a service system through a file uploading method and responding, wherein the called access parameters comprise a file object, a file key, a system id, a system name, a user id, a user name and the like, file operation can be carried out to finish uploading of a file, the file key is used as an identifier for storing the file and is stored in a local database of a client, and operations such as downloading, deleting, address obtaining and the like can be carried out on a specific file through the identifier;
2. when the object storage is adopted, the unified service of the integrated SDK can be called to realize file reading, the calling parameter value can comprise an api, the calling parameter value can comprise a first token character string and a bucket name, the token refers to the meaning of a token or a certificate, and the token is a string of character strings generated by a service end and used as an identifier for a customer terminal to request;
3. generating a second token by using a local method which can be called in the object storage and returning, and taking a bucket name according to an apple when checking a library or caching, wherein the called entry value can comprise a service address, an AK (Access Key ID), an SK (Secret Access Key) and the bucket name, and the called value comprises a second token character string to finish the storage of the file;
4. at a client server, such as in cloud storage, an integrated SDK can be directly called to realize file uploading, the called entry values comprise bucket names, file keys, token, file files and the like, and the called exit values comprise response codes, response messages and the like;
5. when the object storage is adopted, the unified service of the integrated SDK can be called to realize the recording of file information, and the called entry values comprise file size, bucket name, system id, system name, user id, user name, file key and the like, and the entry values comprise operation type description, return codes, return messages, operation time and the like;
6. and an operation log can be recorded in the adopted object storage, and the called entry parameter values comprise a system id, a system name, a user id, a user name, operation type description, a return code, a return message and operation time, so that the operation log is formed.
When a client requires to use a fastdfs technology as a Storage service, a fastdfs Storage cluster is deployed firstly, a Tracker server, a Storage server, a Nginx load balancing server and the like are deployed, a Tracker server address is configured to a service dispatching center, a jar package of the fastdfs client is integrated to the dispatching center, service functions of uploading, downloading, deleting and the like of a file are accessed, and finally service information is configured to a resource pool to realize service starting.
The embodiment can be applied to an Object Storage service, a data path (data reading or writing) and a control path (metadata) can be separated by adopting Object Storage, and a Storage system is built based on Object-based Storage devices (OSD), wherein each Object-based Storage device has certain intelligence and can automatically manage data distribution on the Object-based Storage device. The object storage is an object-based storage device, has intelligent and self-management capabilities, and can realize the reading and writing of objects and the access of storage resources through a web service protocol.
Two data descriptions are contained in the object storage system: the system comprises a container (Bucket) and an Object (Object), wherein the container and the Object both have a globally unique ID, the Object storage adopts a flat structure to manage all Data, and a user/application can access the container/Object and related Data (Data), metadata (metadata) and Object Attribute (Attribute) only according to the ID after the user/application is authenticated by an access code (accessKey). Because the object Storage is based on the object ID, all objects and buckets are managed in a flat mode, data can be directly accessed according to the object ID, the problem that data searching is long in time under the condition of mass data like a complex directory tree structure of a Network Attached Storage (NAS) is solved, and the object Storage has extremely strong expansibility.
Example two
As shown in fig. 3, a multi-source storage device 10 according to the present embodiment is shown, and includes a receiving unit 11, an identifying unit 12, and a processing unit 13, where the receiving unit 11 is configured to receive service information sent by a client and device information of the client, where the service information includes a type and service data, the identifying unit 12 is configured to identify the type of the service information, and determine a service storage mode according to the type of the service information, where each type of the service information corresponds to a service storage mode, and the processing unit 13 is configured to store the service data according to the determined service storage mode.
The multi-source storage device 10 achieves integration of interfaces through the receiving unit 11 to bring convenience to access of files and file streams, hides complexity of accessing specific storage services, reduces learning cost, improves development and production efficiency, can concentrate on realization of self business logic in a business system, does not need to care about construction, use and operation and maintenance of file storage service clusters, achieves one-key switching, is convenient and fast, can improve access performance, reliability and safety of files due to application of a distributed storage architecture, and enables the business system to be more robust.
The general media (devices) of the receiving unit 11 can adopt routers and switches with routing function, soft routing software, etc., the receiving unit 11 sends a data packet from one device to another device in a different network segment, the implementation of routing is implemented by records in the routing table, and the functions that can be implemented by the receiving unit 11 include: separate broadcast domains, select paths to destinations in routing tables, maintain and examine routing information, and connect to wide area networks.
In this embodiment, the type of the service information includes an API call and an SDK call, where the API call corresponds to the first service storage mode, and the SDK call corresponds to the second service storage mode. And selecting an API calling mode or an SDK calling mode to be butted with the unified interface in the processing unit 13 according to the application scene, wherein the API calling mode is served through an REST interface, and the SDK calling mode is served through an integrated jar packet mode, so that files with different sizes and types can be distinguished, stored and optimized for storage service.
In specific practice, when a company business system is connected to each medical insurance office, the first party has different specification requirements on the file storage block, or a business system in a public network environment cannot be used due to field network environment limitation. This requires the development team to study and access various storage services in different environments, which involves a series of matters such as server deployment, code development, resource purchase, etc. The embodiment can systematically package and integrate several mainstream storage services, and access and support more storage sources by continuously collecting the actual requirements of the business system in work, so as to provide services externally through a uniform interface, hide the diversity of various system interfaces, avoid changing business codes even if the storage services are switched, improve the convenience of using object storage services, avoid the need of putting efforts in each system, and prevent the waste of repetitive labor and resources.
EXAMPLE III
As shown in fig. 4, the computer system includes a plurality of computer devices 20, in the second embodiment, components of the multi-source storage apparatus may be distributed in different computer devices 20, and the computer devices 20 may be smartphones, tablet computers, notebook computers, desktop computers, rack-mounted servers, blade servers, tower servers, or rack-mounted servers (including independent servers or a server cluster formed by a plurality of servers) that execute programs, and the like. The computer device 20 of the present embodiment includes at least, but is not limited to: a memory 21 and a processor 22 which are communicatively connected to each other via a system bus. It is noted that fig. 4 only shows the computer device 20 with components 21-22, but it is to be understood that not all shown components are required to be implemented, and that more or fewer components may be implemented instead.
In the present embodiment, the memory 21 (i.e., a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the storage 21 may be an internal storage unit of the computer device 100, such as a hard disk or a memory of the computer device 20. In other embodiments, the memory 21 may also be an external storage device of the computer device 20, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, provided on the computer device 20. Of course, the memory 21 may also include both internal and external storage devices of the computer device 20. In this embodiment, the memory 21 is generally used for storing an operating system and various types of application software installed in the computer system device. Further, the memory 21 may also be used to temporarily store various types of data that have been output or are to be output.
Processor 22 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 22 is typically used to control the overall operation of the computer device 20. In this embodiment, the processor 22 is configured to execute the program code stored in the memory 21 or process data. The processors 22 of the multiple computer devices 20 of the computer system of the present embodiment jointly execute the computer program to implement the multi-source storage method of the first embodiment.
Example four
The present embodiment also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application store, etc., on which a computer program is stored, which when executed by a processor implements corresponding functions. The computer-readable storage medium of this embodiment stores the multi-source storage apparatus 10 of the second embodiment, and when executed by a processor, the computer-readable storage medium implements the multi-source storage method of the first embodiment.
Further, the computer-readable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to the use of the blockchain node, and the like.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one of 8230, and" comprising 8230does not exclude the presence of additional like elements in a process, method, article, or apparatus comprising the element.
The application is operational with numerous general purpose or special purpose computing system environments or configurations. For example: personal computers, server computers, hand-held or portable devices, tablet-type devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
Through the description of the foregoing embodiments, it is clear to those skilled in the art that the method of the foregoing embodiments may be implemented by software plus a necessary general hardware platform, and certainly may also be implemented by hardware, but in many cases, the former is a better implementation.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (8)
1. A multi-source storage method is characterized by comprising the following steps:
s10, receiving service information sent by a client, wherein the service information comprises types and service data, the types of the service information comprise API calls and SDK calls, the API calls correspond to a first service storage mode, and the SDK calls correspond to a second service storage mode;
s20, identifying the type of the service information, and determining a service storage mode according to the type of the service information, wherein each type of the service information corresponds to one service storage mode;
and S30, storing the service data according to the determined service storage mode.
2. The multi-source storage method according to claim 1, wherein the step S10 further includes receiving device information of a client sent by a client, matching the device information of the client with a preset list, and if matching, receiving the service information sent by the client by using a unified interface.
3. The multi-source storage method of claim 2, wherein the preset list includes preset client information;
the matching the device information of the client with a preset list comprises:
matching the equipment information of the client with preset client information in the preset list, and if the equipment information of the client is matched with the preset client information in the preset list, receiving the service information sent by the client by adopting a unified interface;
the preset list is stored in a block chain.
4. The multi-source storage method according to claim 3, wherein the step S30 stores the business data according to the determined business storage mode, and includes:
when the type of the service information is API call, storing the service data by adopting a first service storage mode: decoding the service data to obtain a file name, a file identifier, file data and a type cluster, converting the file name, the file unique identifier, the file data and the type cluster into a file stream, and storing the file stream into a database corresponding to the type cluster, wherein each type cluster corresponds to a corresponding database.
5. The multi-source storage method according to claim 3, wherein the step S30 stores the business data according to the determined business storage mode, and includes:
when the type of the service information is called by the SDK, a second service storage mode is adopted to store the service data: decoding the service data, acquiring file data, a file identifier and a type cluster, inquiring a preset database according to the data volume of the file data, acquiring a database matched with the type cluster, acquiring an address of the database, generating a database identifier corresponding to the database according to the database address, sending the service data to the database according to the database identifier, recording the storage information of the service data, generating a storage log, and sending response information to the client.
6. A multi-source storage device, comprising:
the system comprises a receiving unit, a processing unit and a processing unit, wherein the receiving unit is used for receiving service information sent by a client, the service information comprises types and service data, the types of the service information comprise API calls and SDK calls, the API calls correspond to a first service storage mode, and the SDK calls correspond to a second service storage mode;
the identification unit is used for identifying the types of the service information and determining service storage modes according to the types of the service information, wherein each type of the service information corresponds to one service storage mode;
and the processing unit is used for storing the service data according to the determined service storage mode.
7. A computer system comprising a plurality of computer devices, each computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processors of the plurality of computer devices collectively implement the steps of the method of any one of claims 1 to 5 when the computer program is executed.
8. A computer-readable storage medium comprising a stored data area storing data created according to use of blockchain nodes and a stored program area storing a computer program, wherein the computer program stored by the storage medium when executed by a processor implements the steps of the method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010575391.5A CN111736775B (en) | 2020-06-22 | 2020-06-22 | Multi-source storage method, device, computer system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010575391.5A CN111736775B (en) | 2020-06-22 | 2020-06-22 | Multi-source storage method, device, computer system and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111736775A CN111736775A (en) | 2020-10-02 |
| CN111736775B true CN111736775B (en) | 2023-02-03 |
Family
ID=72652020
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010575391.5A Active CN111736775B (en) | 2020-06-22 | 2020-06-22 | Multi-source storage method, device, computer system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111736775B (en) |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112235409B (en) * | 2020-10-19 | 2023-10-10 | 平安证券股份有限公司 | File uploading method and device, electronic equipment and computer readable storage medium |
| CN113780704A (en) * | 2020-10-22 | 2021-12-10 | 北京京东振世信息技术有限公司 | Task processing method and device |
| CN112416877A (en) * | 2020-11-24 | 2021-02-26 | 武汉联影医疗科技有限公司 | Medical data storage method and device, computer equipment and storage medium |
| CN112433680A (en) * | 2020-11-30 | 2021-03-02 | 中国人寿保险股份有限公司 | Unified interface processing method, device, equipment and medium for multiple storage media |
| CN112667180A (en) * | 2020-12-29 | 2021-04-16 | 百果园技术(新加坡)有限公司 | Business data display method and device, computer equipment and storage medium |
| CN113220759A (en) * | 2021-04-27 | 2021-08-06 | 深圳市云网万店科技有限公司 | Big data storage service sharing method and device, computer equipment and storage medium |
| CN113760868A (en) * | 2021-09-13 | 2021-12-07 | 武汉联影智融医疗科技有限公司 | Data processing method and device and storage service system |
| CN114706919B (en) * | 2022-06-06 | 2022-09-13 | 南京安元科技有限公司 | Distributed storage data storage compatible method and system |
| CN115426322B (en) * | 2022-08-23 | 2023-09-19 | 绿盟科技集团股份有限公司 | Virtual storage method and device |
| CN115545690A (en) * | 2022-11-28 | 2022-12-30 | 云账户技术(天津)有限公司 | Account checking file processing method and device, electronic equipment and readable storage medium |
| CN116541015B (en) * | 2023-05-15 | 2024-03-19 | 上海携宁计算机科技股份有限公司 | Back-end storage method, device, server and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106452815A (en) * | 2015-08-11 | 2017-02-22 | 中移物联网有限公司 | Informatization management method, device and system |
| CN106484329A (en) * | 2016-09-26 | 2017-03-08 | 浪潮电子信息产业股份有限公司 | A kind of big data transmission integrity protection mechanism based on multistage storage |
| CN106933872A (en) * | 2015-12-30 | 2017-07-07 | 阿里巴巴集团控股有限公司 | A kind of method and device that cloud storage service is accessed by traditional file systemses interface |
| CN109634996A (en) * | 2018-10-25 | 2019-04-16 | 深圳壹账通智能科技有限公司 | Customer information table generating method, device, equipment and computer readable storage medium |
| CN111212111A (en) * | 2019-12-17 | 2020-05-29 | 腾讯科技（深圳）有限公司 | Object storage service management method and electronic equipment |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10380026B2 (en) * | 2014-09-04 | 2019-08-13 | Sandisk Technologies Llc | Generalized storage virtualization interface |
| US10609177B2 (en) * | 2018-01-31 | 2020-03-31 | Nutanix, Inc. | Consolidated cloud system for virtual machine |
-
2020
- 2020-06-22 CN CN202010575391.5A patent/CN111736775B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106452815A (en) * | 2015-08-11 | 2017-02-22 | 中移物联网有限公司 | Informatization management method, device and system |
| CN106933872A (en) * | 2015-12-30 | 2017-07-07 | 阿里巴巴集团控股有限公司 | A kind of method and device that cloud storage service is accessed by traditional file systemses interface |
| CN106484329A (en) * | 2016-09-26 | 2017-03-08 | 浪潮电子信息产业股份有限公司 | A kind of big data transmission integrity protection mechanism based on multistage storage |
| CN109634996A (en) * | 2018-10-25 | 2019-04-16 | 深圳壹账通智能科技有限公司 | Customer information table generating method, device, equipment and computer readable storage medium |
| CN111212111A (en) * | 2019-12-17 | 2020-05-29 | 腾讯科技（深圳）有限公司 | Object storage service management method and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111736775A (en) | 2020-10-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111736775B (en) | Multi-source storage method, device, computer system and storage medium | |
| US11683223B2 (en) | Edge database management of the network data plane | |
| US11615084B1 (en) | Unified data processing across streaming and indexed data sets | |
| US10623476B2 (en) | Endpoint management system providing an application programming interface proxy service | |
| US11392416B2 (en) | Automated reconfiguration of real time data stream processing | |
| US10515058B2 (en) | Unified file and object data storage | |
| US9971823B2 (en) | Dynamic replica failure detection and healing | |
| JP2022529967A (en) | Extracting data from the blockchain network | |
| US20220121628A1 (en) | Streaming synthesis of distributed traces from machine logs | |
| US10891569B1 (en) | Dynamic task discovery for workflow tasks | |
| CN107347062A (en) | A kind of method, electronic equipment and the readable storage medium storing program for executing of daily record data processing | |
| US11882154B2 (en) | Template representation of security resources | |
| US20220124110A1 (en) | Anomaly detection using an ensemble of detection models | |
| US11178197B2 (en) | Idempotent processing of data streams | |
| CN113486095A (en) | Civil aviation air traffic control cross-network safety data exchange management platform | |
| US10630582B1 (en) | Scalable border gateway protocol routing | |
| US9503351B1 (en) | Deployment feedback for system updates to resources in private networks | |
| US10567234B1 (en) | Point of origin capture for data compliance management in cloud computing environment | |
| CN112035402A (en) | File storage method and device and terminal equipment | |
| US11106540B1 (en) | Database command replay | |
| US11687487B1 (en) | Text files updates to an active processing pipeline | |
| CN114925044A (en) | Data synchronization method, device and equipment based on cloud storage and storage medium | |
| US11500824B1 (en) | Database proxy | |
| US10346366B1 (en) | Management of a data processing pipeline | |
| US11663058B1 (en) | Preemptive filtering of events of an event bus with a deterministic filter |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220531 Address after: 518000 China Aviation Center 2901, No. 1018, Huafu Road, Huahang community, Huaqiang North Street, Futian District, Shenzhen, Guangdong Province Applicant after: Shenzhen Ping An medical and Health Technology Service Co.,Ltd. Address before: Room 12G, Area H, 666 Beijing East Road, Huangpu District, Shanghai 200001 Applicant before: PING AN MEDICAL AND HEALTHCARE MANAGEMENT Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |