CN110888862A - Data storage method, data query method, data storage device, data query device, server and storage medium - Google Patents
Data storage method, data query method, data storage device, data query device, server and storage medium Download PDFInfo
- Publication number
- CN110888862A CN110888862A CN201911171605.6A CN201911171605A CN110888862A CN 110888862 A CN110888862 A CN 110888862A CN 201911171605 A CN201911171605 A CN 201911171605A CN 110888862 A CN110888862 A CN 110888862A
- Authority
- CN
- China
- Prior art keywords
- data
- text field
- field
- text
- compressed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/211—Schema design and management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a data storage method, a data query method, a data storage device, a data query device, a server and a storage medium. The data storage method comprises the following steps: receiving data to be stored transmitted based on a storage request, wherein the data to be stored comprises one or more text fields, and each text field corresponds to a unique field identifier; judging whether the text field needs to be compressed according to the field identification; if the text field needs to be compressed, calling a preset compression algorithm to compress the text field based on a configured custom processor; and converting the data type of the compressed data to be stored and storing the data to the database. The effect of reducing the occupied space of the disk of the database is achieved.
Description
Technical Field
The embodiment of the invention relates to the technical field of data processing, in particular to a data storage method, a data query method, a data storage device, a data query device, a server and a storage medium.
Background
In the process of project development, it may happen that some relatively large text data (for example, JSON string results returned by a certain interface, message results returned by a certain SDK, and the like) needs to be stored in corresponding fields of the database, and meanwhile, frequent access operations may be subsequently performed on the text data.
Currently, it is common practice to define a large text field type in the database, such as text (65535 characters may be stored) or media text (16777215 characters may be stored), and then to perform read and write operations by conventional methods.
However, storing these large fields directly in the database takes up significant disk space in the database.
Disclosure of Invention
The embodiment of the invention provides a data storage method, a data query method, a data storage device, a data query device, a server and a storage medium, and aims to achieve the effect of reducing the occupied space of a disk of a database.
In a first aspect, an embodiment of the present invention provides a data storage method, including:
receiving data to be stored transmitted based on a storage request, wherein the data to be stored comprises one or more text fields, and each text field corresponds to a unique field identifier;
judging whether the text field needs to be compressed according to the field identification;
if the text field needs to be compressed, calling a preset compression algorithm to compress the text field based on a configured custom processor;
and converting the data type of the compressed data to be stored and storing the data to the database.
Optionally, before the step of calling a preset compression algorithm to compress the text field based on the configured custom processor, the method includes:
and packaging the preset compression algorithm into a custom tool class.
Optionally, the custom processor is configured with a storage address of the preset compression algorithm, and based on the configured custom processor, the preset compression algorithm is called to compress the text field, including:
and calling the preset compression algorithm from the storage address to compress the text field.
Optionally, the determining whether the text field needs to be compressed according to the field identifier includes:
judging whether the field identification is configured with a custom processor or not according to the field identification;
and if the field identification is configured with a custom processor, the text field corresponding to the field identification needs to be compressed.
Optionally, the custom processor is a custom typehandler.
Optionally, the preset compression algorithm includes a GZIP compression algorithm, a deflate compression algorithm, a snappy compression algorithm, or an LZ4 compression algorithm.
In a second aspect, an embodiment of the present invention provides a data query method, including:
receiving data to be queried transmitted based on a query request, wherein the data to be queried comprises one or more text fields, and each text field corresponds to a unique field identifier;
converting the data type of the data to be inquired;
judging whether the text field needs to be decompressed according to the field identification;
if the text field needs to be decompressed, based on a configured custom processor, a preset decompression algorithm is called to decompress the text field;
and sending the decompressed data to be queried to a user for query.
In a third aspect, an embodiment of the present invention provides a data storage device, including:
the receiving module is used for receiving data to be stored transmitted based on the storage request, wherein the data to be stored comprises one or more text fields, and each text field corresponds to a unique field identifier;
the judging module is used for judging whether the text field needs to be compressed according to the field identification;
the compression module is used for calling a preset compression algorithm to compress the text field based on a configured custom processor if the text field needs to be compressed;
and the storage module is used for converting the data type of the compressed data to be stored and storing the data to the database.
In a fourth aspect, an embodiment of the present invention provides a data query apparatus, including:
the receiving module is used for receiving data to be inquired transmitted based on the inquiry request, wherein the data to be inquired comprises one or more text fields, and each text field corresponds to a unique field identifier;
the data type conversion module is used for converting the data type of the data to be inquired;
the judging module is used for judging whether the text field needs to be decompressed according to the field identification;
the decompression module is used for calling a preset decompression algorithm to decompress the text field based on a configured custom processor if the text field needs to be decompressed;
and the sending module is used for sending the decompressed data to be queried to a user for querying.
In a fifth aspect, an embodiment of the present invention provides a server, including:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement a data storage method or a data query method as described in any embodiment of the invention.
In a sixth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a data storage method or a data query method according to any embodiment of the present invention.
The embodiment of the invention receives the data to be stored transmitted based on the storage request, wherein the data to be stored comprises one or more text fields, and each text field corresponds to a unique field identifier; judging whether the text field needs to be compressed according to the field identification; if the text field needs to be compressed, calling a preset compression algorithm to compress the text field based on a configured custom processor; the compressed data to be stored is converted in data type and then stored in the database, so that the problem that the large fields are directly stored in the database and occupy the disk space of the database is solved, and the effect of reducing the occupied space of the disk of the database is realized.
Drawings
Fig. 1 is a schematic flow chart of a data storage method according to an embodiment of the present invention;
fig. 2 is a schematic flowchart of a data query method according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a data storage device according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of a data query device according to a fourth embodiment of the present invention;
fig. 5 is a schematic structural diagram of a server according to a fifth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the steps as a sequential process, many of the steps can be performed in parallel, concurrently or simultaneously. In addition, the order of the steps may be rearranged. A process may be terminated when its operations are completed, but may have additional steps not included in the figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.
Furthermore, the terms "first," "second," and the like may be used herein to describe various orientations, actions, steps, elements, or the like, but the orientations, actions, steps, or elements are not limited by these terms. These terms are only used to distinguish one direction, action, step or element from another direction, action, step or element. For example, the first information may be referred to as second information, and similarly, the second information may be referred to as first information, without departing from the scope of the present application. The first information and the second information are both information, but they are not the same information. The terms "first", "second", etc. are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
Example one
Fig. 1 is a schematic flowchart of a data storage method according to an embodiment of the present invention, which is applicable to a scenario in which data is compressed and then stored in a database.
As shown in fig. 1, a data storage method provided in an embodiment of the present invention includes:
s110, receiving data to be stored transmitted based on a storage request, wherein the data to be stored comprises one or more text fields, and each text field corresponds to a unique field identifier.
The storage request is a request generated when data needs to be stored. The data to be stored refers to data which needs to be stored in a database. Specifically, the data type of the data to be stored is a JAVA data type. The text field refers to a piece of text describing information, such as a name, an age, a JSON (JSON object notation) character string result returned by a certain interface, a message result returned by a certain SDK (Software Development Kit), and the like, and is not limited herein. Specifically, each text field corresponds to a unique field identifier, and the field identifier is an identifier used for determining the specific text field. In this embodiment, optionally, the field identifier may be a field name (e.g., JSON) of the text field, and may also be other identifiers having uniqueness, which is not limited herein.
And S120, judging whether the text field needs to be compressed according to the field identification.
The field identification has uniqueness, so that which text fields need to be compressed and which text fields do not need to be compressed can be judged according to the field identification. Specifically, the field identifier to be compressed may be configured in advance, and which text is to be compressed may be determined according to the field identifier, so that the compression is performed in a targeted manner. If the field identification is configured with a compression function in advance, the text field corresponding to the field identification needs to be compressed. And if the text field needs to be compressed, executing the step S130, and calling a preset compression algorithm to compress the text field based on the configured custom processor. Specifically, during the development of a project, a lot of data to be stored is generated. Some text fields have less data, the repeated data is less, and the compression function is not required to be configured for the text fields; for some text fields with more data (such as returned JSON string results), more repeated data is needed, and in order to reduce the occupied space of the disk, a compression function needs to be configured for the text fields. For example, if a text field exceeds 1MB (megabyte), there must be a lot of data that is repeatedly compressible, and the text field may be configured with a compression function to compress the text field before storing the text field in the database.
In an embodiment, optionally, determining whether the text field needs to be compressed according to the field identifier may include:
judging whether the field identification is configured with a custom processor or not according to the field identification;
and if the field identification is configured with a custom processor, the text field corresponding to the field identification needs to be compressed.
In this embodiment, a custom processor may be configured on the field identifier, and then the custom processor of the field identifier is configured, and the text field corresponding to the field identifier needs to be compressed. The embodiment configures the custom processor by the field identifier, that is, configures the custom processor on the text field. In the mybatis open source framework, when data to be stored is stored, the data type must be converted by a typehandler carried by the mybatis open source framework and then stored in a database. And by configuring the custom processor on the field identifier, the characteristic that the self-contained processor can convert the data type in the open source framework is utilized, the custom processor is configured, and the compression function is executed before the data type is converted.
And S130, calling a preset compression algorithm to compress the text field based on the configured custom processor.
The custom processor refers to a processor improved for the processor with the self-contained open source framework, and the name of the custom processor is different from that of the processor with the self-contained open source framework. For example, if the processor of the mybatis framework is typehandler, the name of the custom processor for compression may be compressshandler, and is not limited herein. In this embodiment, the custom processor is a processor configured with a compression function for a processor provided in the open source framework, and the custom processor compresses a part of text fields in data to be stored, and converts the compressed text fields into a database data format for storage. Alternatively, the open source framework may be a mybatis framework or hibernate framework, and the like, without limitation. For example, when the open source framework is a mybatis framework, the processor of the open source framework is typehandler, and the processor is used as a converter for JAVA data types and database data types. The preset compression algorithm refers to an algorithm for compressing a text field. Optionally, the preset compression algorithm includes, but is not limited to, a GZIP compression algorithm, a deflate compression algorithm, a snappy compression algorithm, or an LZ4 compression algorithm, and is not limited herein.
Specifically, when the mybatis frame is used for storing data to be stored, a typehandler carried in the mybatis frame is required to convert a JAVA data type into a database data type, and then the JAVA data type is stored in the database.
In an embodiment, optionally, the custom processor is configured with a storage address of the preset compression algorithm, and the invoking the preset compression algorithm to compress the text field based on the configured custom processor may include:
and calling the preset compression algorithm from the storage address to compress the text field.
The storage address refers to an address for storing a code of a preset compression algorithm. And when the text field needs to be compressed, calling a preset compression algorithm to compress the text field according to the configured storage address in the custom processor. Illustratively, when the text field A needs to be compressed, a preset compression algorithm is called to compress the text field A, and the JAVA data type is directly converted into the database data type without the need of the compressed text field.
And S140, converting the data type of the compressed data to be stored and storing the data to a database.
After all or part of text fields in the data to be stored are compressed, the compressed data to be stored are converted from the JAVA data type to the database type and then are stored in the database for storage.
In the embodiment, the data type is converted after the text field is compressed, so that the occupied space of the database is reduced. In addition, the efficiency of reading the disk can be improved. The compression process is carried out in the JAVA application program, so that the compression performance is better. In addition, only a corresponding custom processor (such as custom typehandler) needs to be configured for the compressed text field, and the text field which does not need to be compressed does not need to be processed, so that the method is less invasive to the existing code.
In an embodiment, optionally, before the step S130, based on the configured custom processor, invoking a preset compression algorithm to compress the text field, the method may include:
and packaging the preset compression algorithm into a custom tool class.
The custom tool class is a tool class using a related function. In this embodiment, the preset compression algorithm is packaged into a custom tool class, i.e. a compressed tool class. The preset compression algorithm is packaged into the custom tool class, so that the details (such as compressed codes) of the preset compression algorithm are hidden, and the preset compression algorithm is directly used when compression is needed, so that the use convenience is improved.
According to the technical scheme of the embodiment of the invention, the data to be stored transmitted based on the storage request is received, the data to be stored comprises one or more text fields, and each text field corresponds to a unique field identifier;
judging whether the text field needs to be compressed according to the field identification; if the text field needs to be compressed, calling a preset compression algorithm to compress the text field based on a configured custom processor; the compressed data to be stored is subjected to data type conversion and then stored in the database, the text field is compressed before being stored in the database, the size of the compressed data to be stored is smaller than that of uncompressed data to be stored, the occupied space is smaller, and the technical effect of reducing the occupied space of a disk of the database is achieved.
Example two
Fig. 2 is a schematic flowchart of a data query method according to a second embodiment of the present invention, which is suitable for a scenario in which data to be queried stored in a database is queried. The method can be executed by a data query device, which can be implemented in software and/or hardware and can be integrated on a server.
As shown in fig. 2, the data query method provided by the second embodiment of the present invention includes:
s210, receiving data to be inquired transmitted based on the inquiry request, wherein the data to be inquired comprises one or more text fields, and each text field corresponds to a unique field identifier.
The query request refers to a request generated when data needs to be queried. The data to be queried refers to data needing to be queried. Specifically, the data to be queried is stored in a database and transmitted from the database to the JAVA application. The data type of the data to be queried is the data type of the database. The text field refers to a piece of text for describing information, such as a name, an age of a user, a JSON (JSON Object Notation) character string result returned by an interface, a message result returned by an SDK (Software Development Kit), and the like, and is not limited herein. Specifically, each text field corresponds to a unique field identifier, and the field identifier is an identifier used for determining the specific text field. In this embodiment, optionally, the field identifier may be a field name (e.g., JSON) of the text field, and may also be other identifiers having uniqueness, which is not limited herein.
And S220, converting the data type of the data to be queried.
The data to be queried stored in the database is the data type of the database, so the data type of the data to be queried needs to be converted. Specifically, the data to be queried is converted from the data type of the database into the JAVA data type, so as to satisfy the use query in the JAVA application.
And S230, judging whether the text field needs to be decompressed according to the field identification.
The field identifier has uniqueness, so that which text fields need to be decompressed and which text fields do not need to be decompressed can be judged according to the field identifier. Specifically, the field identifier that needs to be decompressed can be configured in advance, and which text needs to be decompressed can be determined according to the field identifier, so that decompression can be performed in a targeted manner. If the field identification is configured with the decompression function in advance, the text field corresponding to the field identification needs to be decompressed. If the text field needs to be decompressed, step S240 is performed. Specifically, decompression is required for text fields that are decompressed for storage in the database.
In an embodiment, optionally, determining whether the text field needs to be decompressed according to the field identifier may include:
judging whether the field identification is configured with a custom processor or not according to the field identification;
and if the field identification is configured with a custom processor, decompressing the text field corresponding to the field identification.
In this embodiment, a custom processor may be configured on the field identifier, and then the custom processor configured with the field identifier needs to decompress a text field corresponding to the field identifier. The embodiment configures the custom processor by the field identifier, that is, configures the custom processor on the text field. In the mybatis open source framework, when data to be queried is queried, the data type needs to be converted by a typehandler provided by the mybatis open source framework, and then the query is performed. And by configuring the custom processor on the field identifier, the characteristic that the self-contained processor can convert the data type in the open source framework is utilized, the custom processor is configured, and the decompression function is executed after the data type is converted.
S240, based on the configured custom processor, calling a preset decompression algorithm to decompress the text field.
The custom processor refers to a processor improved for the processor with the self-contained open source framework, and the name of the custom processor is different from that of the processor with the self-contained open source framework. For example, if the processor of the mybatis framework is typehandler, the name of the custom processor used for decompression may be decompenssharandler, and is not limited herein. In this embodiment, the custom processor is a processor configured with a decompression function for a processor provided with an open source framework, and decompresses a part of the text field by using the custom processor after converting the data type of the data to be queried by using the processor provided with the open source framework. Alternatively, the open source framework may be a mybatis framework or hibernate framework, and the like, without limitation. For example, when the open source framework is a mybatis framework, the processor of the open source framework is typehandler, and the processor is used as a converter for JAVA data types and database data types. The preset decompression algorithm refers to an algorithm for decompressing a text field. Optionally, the preset decompression algorithm includes, but is not limited to, a GZIP decompression algorithm, a deflate decompression algorithm, a snappy decompression algorithm, or an LZ4 decompression algorithm, and is not limited herein.
Specifically, when a mybatis framework is used for querying a query, a typehandler carried in the mybatis framework is required to convert the data type of the database into a JAVA data type, and then a user-defined processor is used for decompression.
In an embodiment, optionally, the configuring, by the custom processor, a storage address of the preset decompression algorithm, and the invoking, by the configured custom processor, the preset decompression algorithm to decompress the text field may include:
and calling the preset decompression algorithm from the storage address to decompress the text field.
The storage address refers to an address for storing a code of a preset decompression algorithm. When the text field needs to be decompressed, a preset decompression algorithm is called to decompress the text field according to the configured storage address in the user-defined processor. Illustratively, when the text field a needs to be decompressed, a preset decompression algorithm is called to decompress the text field a, and the data type of the database is directly converted into the JAVA data type without the decompressed text field.
And S250, sending the decompressed data to be queried to a user for query.
After all or part of text fields in the data to be queried are decompressed, the decompressed data to be queried is sent to a user for query.
In this embodiment, the data to be queried is stored in the database, and a part of the text field is compressed, so that the occupied space of the database is reduced, and when the query is needed, the query can be performed by decompressing the data to be queried. In addition, the efficiency of reading the disk can be improved. The decompression process is carried out in the JAVA application program, so that the decompression performance is better. In addition, only a corresponding custom processor (such as custom typehandler) needs to be configured for the decompressed text field, and the text field which does not need decompression processing does not need to be processed, so that the method is less invasive to the existing code.
In an embodiment, optionally, before invoking a preset decompression algorithm to decompress the text field based on the configured custom processor in step S240, the method may include:
and packaging the preset decompression algorithm into a user-defined tool class.
The custom tool class is a tool class using a related function. In this embodiment, the preset decompression algorithm is packaged into a custom tool class, i.e. a decompressed tool class. The preset decompression algorithm is packaged into the user-defined tool class, so that the details (such as decompressed codes) of the preset decompression algorithm are hidden, the preset decompression algorithm can be directly used when decompression is needed, and the convenience in use is improved.
According to the technical scheme of the embodiment of the invention, the data to be inquired transmitted based on the inquiry request is received, the data to be inquired comprises one or more text fields, and each text field corresponds to a unique field identifier; converting the data type of the data to be inquired; judging whether the text field needs to be decompressed according to the field identification; if the text field needs to be decompressed, based on a configured custom processor, a preset decompression algorithm is called to decompress the text field; the decompressed data to be inquired is sent to a user for inquiry, and the data to be inquired stored in the database is compressed, so that the technical effect of reducing the occupied space of a disk of the database is achieved.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a data storage device according to a third embodiment of the present invention, where this embodiment is applicable to a scenario where data is compressed and then stored in a database, and the data storage device may be implemented in a software and/or hardware manner and may be integrated on a server.
As shown in fig. 3, the data storage apparatus provided in this embodiment may include a receiving module 310, a determining module 320, a compressing module 330, and a storing module 340, where:
a receiving module 310, configured to receive data to be stored, which is transmitted based on a storage request, where the data to be stored includes one or more text fields, and each text field corresponds to a unique field identifier;
the judging module 320 is configured to judge whether the text field needs to be compressed according to the field identifier;
the compression module 330 is configured to, if the text field needs to be compressed, invoke a preset compression algorithm to compress the text field based on a configured custom processor;
the storage module 340 is configured to perform data type conversion on the compressed data to be stored, and store the data to the database.
Optionally, the apparatus further comprises:
and the packaging module is used for packaging the preset compression algorithm into a user-defined tool class.
The custom processor is configured with a storage address of the preset compression algorithm, and the compression module 330 is specifically configured to invoke the preset compression algorithm from the storage address to compress the text field.
Optionally, the determining module 320 includes:
the judging unit is used for judging whether the field identification is configured with a custom processor or not according to the field identification; and if the field identification is configured with a custom processor, the text field corresponding to the field identification needs to be compressed.
Optionally, the custom processor is a custom typehandler.
Optionally, the preset compression algorithm includes a GZIP compression algorithm, a deflate compression algorithm, a snappy compression algorithm, or an LZ4 compression algorithm.
The data storage device provided by the embodiment of the invention can execute the data storage method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method. Reference may be made to the description of any method embodiment of the invention not specifically described in this embodiment.
Example four
Fig. 4 is a schematic structural diagram of a data query device according to a fourth embodiment of the present invention, where this embodiment is applicable to a scenario where data to be queried stored in a database is queried, and the device may be implemented in a software and/or hardware manner and may be integrated on a server.
As shown in fig. 4, the data storage apparatus provided in this embodiment may include a receiving module 410, a data type converting module 420, a determining module 430, a decompressing module 440, and a sending module 450, where:
a receiving module 410, configured to receive data to be queried transmitted based on a query request, where the data to be queried includes one or more text fields, and each text field corresponds to a unique field identifier;
a data type conversion module 420, configured to perform data type conversion on the data to be queried;
the judging module 430 is configured to judge whether the text field needs to be decompressed according to the field identifier;
the decompression module 440 is configured to invoke a preset decompression algorithm to decompress the text field based on a configured custom processor if the text field needs to be decompressed;
the sending module 450 is configured to send the decompressed data to be queried to a user for querying.
Optionally, the determining module 430 includes:
the judging unit is used for judging whether the field identification is configured with a custom processor or not according to the field identification; and if the field identification is configured with a custom processor, decompressing the text field corresponding to the field identification.
Optionally, the custom processor is configured with a storage address of the preset decompression algorithm, and the decompression module 440 is specifically configured to invoke the preset decompression algorithm from the storage address to decompress the text field.
Optionally, the apparatus further comprises:
and the packaging module is used for packaging the preset decompression algorithm into a user-defined tool class.
Optionally, the custom processor is a custom typehandler.
Optionally, the preset decompression algorithm includes a GZIP decompression algorithm, a deflate decompression algorithm, a snappy decompression algorithm, or an LZ4 decompression algorithm.
The data query device provided by the embodiment of the invention can execute the data query method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method. Reference may be made to the description of any method embodiment of the invention not specifically described in this embodiment.
EXAMPLE five
Fig. 5 is a schematic structural diagram of a server according to a fifth embodiment of the present invention. FIG. 5 illustrates a block diagram of an exemplary server 612 suitable for use in implementing embodiments of the invention. The server 612 shown in fig. 5 is only an example, and should not bring any limitation to the function and the scope of the use of the embodiments of the present invention.
As shown in fig. 5, the server 612 is in the form of a general-purpose server. The components of server 612 may include, but are not limited to: one or more processors 616, a memory device 628, and a bus 618 that couples the various system components including the memory device 628 and the processors 616.
The server 612 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by server 612 and includes both volatile and nonvolatile media, removable and non-removable media.
A program/utility 640 having a set (at least one) of program modules 642 may be stored, for example, in storage 628, such program modules 642 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. The program modules 642 generally perform the functions and/or methods of the described embodiments of the present invention.
The server 612 may also communicate with one or more external devices 614 (e.g., keyboard, pointing terminal, display 624, etc.), with one or more terminals that enable a user to interact with the server 612, and/or with any terminals (e.g., network card, modem, etc.) that enable the server 612 to communicate with one or more other computing terminals. Such communication may occur via input/output (I/O) interfaces 622. Further, server 612 may communicate with one or more networks (e.g., a Local Area Network (LAN), Wide Area Network (WAN), and/or a public Network such as the internet) via Network adapter 620. As shown in FIG. 5, the network adapter 620 communicates with the other modules of the server 612 via the bus 618. It should be appreciated that although not shown, other hardware and/or software modules may be used in conjunction with the server 612, including but not limited to: microcode, end drives, Redundant processors, external disk drive Arrays, RAID (Redundant Arrays of Independent Disks) systems, tape drives, and data backup storage systems, among others.
The processor 616 executes various functional applications and data processing by executing programs stored in the storage device 628, for example, implementing a data storage method provided by any embodiment of the present invention, which may include:
receiving data to be stored transmitted based on a storage request, wherein the data to be stored comprises one or more text fields, and each text field corresponds to a unique field identifier;
judging whether the text field needs to be compressed according to the field identification;
if the text field needs to be compressed, calling a preset compression algorithm to compress the text field based on a configured custom processor;
and converting the data type of the compressed data to be stored and storing the data to the database.
Or implement a data query method provided by any embodiment of the present invention, the method may include:
receiving data to be queried transmitted based on a query request, wherein the data to be queried comprises one or more text fields, and each text field corresponds to a unique field identifier;
converting the data type of the data to be inquired;
judging whether the text field needs to be decompressed according to the field identification;
if the text field needs to be decompressed, based on a configured custom processor, a preset decompression algorithm is called to decompress the text field;
and sending the decompressed data to be queried to a user for query.
According to the technical scheme of the embodiment of the invention, the data to be stored transmitted based on the storage request is received, the data to be stored comprises one or more text fields, and each text field corresponds to a unique field identifier; judging whether the text field needs to be compressed according to the field identification; if the text field needs to be compressed, calling a preset compression algorithm to compress the text field based on a configured custom processor; converting the data type of the compressed data to be stored, and storing the data to a database, or receiving the data to be queried transmitted based on a query request, wherein the data to be queried comprises one or more text fields, and each text field corresponds to a unique field identifier; converting the data type of the data to be inquired; judging whether the text field needs to be decompressed according to the field identification; if the text field needs to be decompressed, based on a configured custom processor, a preset decompression algorithm is called to decompress the text field; and sending the decompressed data to be queried to a user for query. The technical effect of reducing the occupied space of the disk of the database is achieved.
EXAMPLE six
An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a data storage method provided in any embodiment of the present invention, and the method may include:
receiving data to be stored transmitted based on a storage request, wherein the data to be stored comprises one or more text fields, and each text field corresponds to a unique field identifier;
judging whether the text field needs to be compressed according to the field identification;
if the text field needs to be compressed, calling a preset compression algorithm to compress the text field based on a configured custom processor;
and converting the data type of the compressed data to be stored and storing the data to the database.
Or implement a data query method provided by any embodiment of the present invention, the method may include:
receiving data to be queried transmitted based on a query request, wherein the data to be queried comprises one or more text fields, and each text field corresponds to a unique field identifier;
converting the data type of the data to be inquired;
judging whether the text field needs to be decompressed according to the field identification;
if the text field needs to be decompressed, based on a configured custom processor, a preset decompression algorithm is called to decompress the text field;
and sending the decompressed data to be queried to a user for query.
The computer-readable storage media of embodiments of the invention may take any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or terminal. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
According to the technical scheme of the embodiment of the invention, the data to be stored transmitted based on the storage request is received, the data to be stored comprises one or more text fields, and each text field corresponds to a unique field identifier; judging whether the text field needs to be compressed according to the field identification; if the text field needs to be compressed, calling a preset compression algorithm to compress the text field based on a configured custom processor; converting the data type of the compressed data to be stored, and storing the data to a database, or receiving the data to be queried transmitted based on a query request, wherein the data to be queried comprises one or more text fields, and each text field corresponds to a unique field identifier; converting the data type of the data to be inquired; judging whether the text field needs to be decompressed according to the field identification; if the text field needs to be decompressed, based on a configured custom processor, a preset decompression algorithm is called to decompress the text field; and sending the decompressed data to be queried to a user for query. The technical effect of reducing the occupied space of the disk of the database is achieved.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (11)
1. A method of storing data, comprising:
receiving data to be stored transmitted based on a storage request, wherein the data to be stored comprises one or more text fields, and each text field corresponds to a unique field identifier;
judging whether the text field needs to be compressed according to the field identification;
if the text field needs to be compressed, calling a preset compression algorithm to compress the text field based on a configured custom processor;
and converting the data type of the compressed data to be stored and storing the data to the database.
2. The data storage method of claim 1, wherein before the step of invoking a preset compression algorithm to compress the text field based on the configured custom processor, the method comprises:
and packaging the preset compression algorithm into a custom tool class.
3. The data storage method of claim 1, wherein the custom processor is configured with a storage address of the preset compression algorithm, and the invoking of the preset compression algorithm to compress the text field based on the configured custom processor comprises:
and calling the preset compression algorithm from the storage address to compress the text field.
4. The data storage method of claim 1, wherein the determining whether the text field needs to be compressed according to the field identifier comprises:
judging whether the field identification is configured with a custom processor or not according to the field identification;
and if the field identification is configured with a custom processor, the text field corresponding to the field identification needs to be compressed.
5. The data storage method of claim 1, wherein the custom processor is a custom typehandler.
6. The data storage method of claim 1, wherein the preset compression algorithm comprises a GZIP compression algorithm, a deflate compression algorithm, a snap compression algorithm, or an LZ4 compression algorithm.
7. A method for querying data, comprising:
receiving data to be queried transmitted based on a query request, wherein the data to be queried comprises one or more text fields, and each text field corresponds to a unique field identifier;
converting the data type of the data to be inquired;
judging whether the text field needs to be decompressed according to the field identification;
if the text field needs to be decompressed, based on a configured custom processor, a preset decompression algorithm is called to decompress the text field;
and sending the decompressed data to be queried to a user for query.
8. A data storage device, comprising:
the receiving module is used for receiving data to be stored transmitted based on the storage request, wherein the data to be stored comprises one or more text fields, and each text field corresponds to a unique field identifier;
the judging module is used for judging whether the text field needs to be compressed according to the field identification;
the compression module is used for calling a preset compression algorithm to compress the text field based on a configured custom processor if the text field needs to be compressed;
and the storage module is used for converting the data type of the compressed data to be stored and storing the data to the database.
9. A data query apparatus, comprising:
the receiving module is used for receiving data to be inquired transmitted based on the inquiry request, wherein the data to be inquired comprises one or more text fields, and each text field corresponds to a unique field identifier;
the data type conversion module is used for converting the data type of the data to be inquired;
the judging module is used for judging whether the text field needs to be decompressed according to the field identification;
the decompression module is used for calling a preset decompression algorithm to decompress the text field based on a configured custom processor if the text field needs to be decompressed;
and the sending module is used for sending the decompressed data to be queried to a user for querying.
10. A server, comprising:
one or more processors;
storage means for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the data storage method of any one of claims 1-6 or the data query method of claim 7.
11. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a data storage method according to any one of claims 1 to 6 or a data query method according to claim 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911171605.6A CN110888862A (en) | 2019-11-26 | 2019-11-26 | Data storage method, data query method, data storage device, data query device, server and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911171605.6A CN110888862A (en) | 2019-11-26 | 2019-11-26 | Data storage method, data query method, data storage device, data query device, server and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110888862A true CN110888862A (en) | 2020-03-17 |
Family
ID=69748732
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911171605.6A Pending CN110888862A (en) | 2019-11-26 | 2019-11-26 | Data storage method, data query method, data storage device, data query device, server and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110888862A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111737404A (en) * | 2020-06-26 | 2020-10-02 | 豆盟(北京)科技股份有限公司 | Data processing method, device, equipment and computer storage medium |
| CN111767280A (en) * | 2020-04-17 | 2020-10-13 | 北京沃东天骏信息技术有限公司 | Data processing method, device and storage medium |
| CN112182085A (en) * | 2020-09-28 | 2021-01-05 | 中国建设银行股份有限公司 | Data export method, device, equipment and storage medium |
| CN112306975A (en) * | 2020-11-06 | 2021-02-02 | 山东云海国创云计算装备产业创新中心有限公司 | File storage method, device, equipment and computer readable storage medium |
| CN113392240A (en) * | 2021-06-11 | 2021-09-14 | 江苏云从曦和人工智能有限公司 | Biological characteristic storage optimization method and device, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104462141A (en) * | 2013-09-24 | 2015-03-25 | ***通信集团重庆有限公司 | Data storage and query method and system and storage engine device |
| CN106934066A (en) * | 2017-03-31 | 2017-07-07 | 联想(北京)有限公司 | A kind of metadata processing method, device and storage device |
| CN108268658A (en) * | 2018-02-06 | 2018-07-10 | 政采云有限公司 | Appendchunk storage method and system based on MySQL database |
| CN109101504A (en) * | 2017-06-20 | 2018-12-28 | 恒为科技（上海）股份有限公司 | A kind of efficient log compression and indexing means |
-
2019
- 2019-11-26 CN CN201911171605.6A patent/CN110888862A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104462141A (en) * | 2013-09-24 | 2015-03-25 | ***通信集团重庆有限公司 | Data storage and query method and system and storage engine device |
| CN106934066A (en) * | 2017-03-31 | 2017-07-07 | 联想(北京)有限公司 | A kind of metadata processing method, device and storage device |
| CN109101504A (en) * | 2017-06-20 | 2018-12-28 | 恒为科技（上海）股份有限公司 | A kind of efficient log compression and indexing means |
| CN108268658A (en) * | 2018-02-06 | 2018-07-10 | 政采云有限公司 | Appendchunk storage method and system based on MySQL database |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111767280A (en) * | 2020-04-17 | 2020-10-13 | 北京沃东天骏信息技术有限公司 | Data processing method, device and storage medium |
| CN111737404A (en) * | 2020-06-26 | 2020-10-02 | 豆盟(北京)科技股份有限公司 | Data processing method, device, equipment and computer storage medium |
| CN112182085A (en) * | 2020-09-28 | 2021-01-05 | 中国建设银行股份有限公司 | Data export method, device, equipment and storage medium |
| CN112306975A (en) * | 2020-11-06 | 2021-02-02 | 山东云海国创云计算装备产业创新中心有限公司 | File storage method, device, equipment and computer readable storage medium |
| CN113392240A (en) * | 2021-06-11 | 2021-09-14 | 江苏云从曦和人工智能有限公司 | Biological characteristic storage optimization method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110888862A (en) | Data storage method, data query method, data storage device, data query device, server and storage medium | |
| CN111090628A (en) | Data processing method and device, storage medium and electronic equipment | |
| CN111008230B (en) | Data storage method, device, computer equipment and storage medium | |
| CN110263277B (en) | Page data display method, page data updating device, page data equipment and storage medium | |
| CN112860798B (en) | Data processing method and device, electronic equipment and storage medium | |
| CN110647316B (en) | Method and device for generating universal business object, computer equipment and storage medium | |
| CN110688096B (en) | Method and device for constructing application program containing plug-in, medium and electronic equipment | |
| CN114528044B (en) | Interface calling method, device, equipment and medium | |
| CN111694866A (en) | Data searching and storing method, data searching system, data searching device, data searching equipment and data searching medium | |
| US20220182072A1 (en) | Data Compression Method and Apparatus, Computer-Readable Storage Medium, and Electronic Device | |
| CN109218393B (en) | Push implementation method, device, equipment and computer storage medium | |
| CN110647753B (en) | Method, device and equipment for acquiring kernel file and storage medium | |
| CN110889880A (en) | Map processing method, device, equipment and storage medium | |
| CN107992457B (en) | Information conversion method, device, terminal equipment and storage medium | |
| CN111835686A (en) | Data compression method, data decompression method, data compression device, data decompression device, electronic equipment and storage medium | |
| CN116561202A (en) | Method and device for serializing object | |
| US11797277B2 (en) | Neural network model conversion method server, and storage medium | |
| CN113849449A (en) | Communication system and information interaction method, device and medium | |
| CN107835158B (en) | Method and device for transmitting interactive data under SNA network system | |
| CN111080728A (en) | Map processing method, device, equipment and storage medium | |
| CN114124883A (en) | Data access method and device based on cloud storage address, computer equipment and medium | |
| CN112748930A (en) | Compilation detection method, device, equipment and storage medium | |
| CN112182085A (en) | Data export method, device, equipment and storage medium | |
| CN111651188A (en) | Data result determining method, device, equipment and storage medium for differential packet | |
| CN113495651A (en) | Window control method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200317 |