CN112579650A - Data processing method and system based on Redis cache - Google Patents
Data processing method and system based on Redis cache Download PDFInfo
- Publication number
- CN112579650A CN112579650A CN202011564956.6A CN202011564956A CN112579650A CN 112579650 A CN112579650 A CN 112579650A CN 202011564956 A CN202011564956 A CN 202011564956A CN 112579650 A CN112579650 A CN 112579650A
- Authority
- CN
- China
- Prior art keywords
- query
- redis
- key
- request
- write
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 41
- 238000012545 processing Methods 0.000 claims abstract description 47
- 238000004422 calculation algorithm Methods 0.000 claims description 8
- 230000000977 initiatory effect Effects 0.000 claims description 3
- 238000012423 maintenance Methods 0.000 abstract description 9
- 238000011161 development Methods 0.000 abstract description 8
- 238000011160 research Methods 0.000 abstract description 6
- 238000000034 method Methods 0.000 abstract description 5
- 238000004590 computer program Methods 0.000 description 9
- 238000003860 storage Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000000926 separation method Methods 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 4
- 238000009826 distribution Methods 0.000 description 4
- 230000008030 elimination Effects 0.000 description 3
- 238000003379 elimination reaction Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Computing Systems (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a data processing method and a system based on Redis cache, wherein the data processing method based on Redis cache comprises the following steps: a Redis cluster building step, which is used for building a Redis cluster, configuring a load balancing strategy and configuring a Redis registry; a data writing step, configured to obtain a write request of a client, write the write request of the client into a database through at least one application instance, perform hotspot data processing on a write key of the write request, and write the write key of the write request into a master _ key of each Redis node in the Redis cluster; and a data query step, configured to obtain a query request of the client and forward the query request to the Redis node and/or the database through the application instance based on the load balancing policy to obtain a query result. By the method and the device, complexity of Redis cluster configuration is reduced, and research, development, operation and maintenance cost is reduced.
Description
Technical Field
The present application relates to the field of internet technologies, and in particular, to a data processing method and system based on a Redis cache, a computer device, and a computer-readable storage medium.
Background
The current internet environment is often accompanied by a scene of high concurrent requests, and when the user volume of the system is accumulated to a certain magnitude, IO reading and writing often become a key for influencing the system performance. The hot data is written into the cache, namely a buffer layer is added between the request and the storage system, IO operation of the relational database is reduced, and system performance can be remarkably improved. Most of the current caching scenes are realized based on third-party middleware. The Redis is a member of the military of the caching technology, and is widely used in the caching field by virtue of characteristics of abundant data structures, thread safety, elimination strategies, snapshots and the like.
The existing Redis cluster cache implementation scheme is realized based on master and slave nodes, sentinel modes and hash grooves. When the data volume is continuously increased, the performance of a single node reaches a bottleneck, and a child node needs to be added to expand the cluster range. The scheme can realize the use and the expansion of the Redis cluster and can be used as a production scheme. But the problems are also obvious, namely the configuration is complicated, the division of node responsibilities is fine, and the link tracking is complex.
The technical scheme for realizing the clustering based on the master-slave nodes, the sentinel model and the hash grooves mainly comprises the steps of carrying out virtual slot position calculation on inquired keys to find out child nodes, wherein the child nodes are formed by combining the master-slave nodes, respectively realize writing and reading operations, and correspondingly update hot data of the slot points. In the technical scheme, the monitoring and the automatic fault switching of the master node and the slave node are realized by sentinels, so that the node allocation has excessive responsibility and high development and maintenance cost, and is not suitable for small systems. The client requests of the technical scheme of realizing the cluster based on the read-write separation mode are issued to each service instance through load balancing, the hot spot data are calculated by using an independent module, and are respectively cached in each service instance and the Redis slave node, so that the same hot spot data are cached at multiple positions, and the maintenance cost is high; caching hot spot data at a service instance also consumes service system resources.
Disclosure of Invention
The embodiment of the application provides a high-availability, expandable and simplified data processing method, system, computer equipment and computer readable storage medium based on Redis cache, reduces complexity of Redis cluster configuration, and reduces research, development, operation and maintenance costs.
In a first aspect, an embodiment of the present application provides a data processing method based on a Redis cache, including:
a Redis cluster building step, which is used for building a Redis cluster, configuring a load balancing strategy and configuring a Redis registry;
a data writing step, configured to obtain a write request of a client, write the write request of the client into a database through at least one application instance, perform hotspot data processing on a write key of the write request, and write the write key of the write request into a master _ key of each Redis node in the Redis cluster;
and a data query step, configured to obtain a query request of the client and forward the query request to the Redis node and/or the database through the application instance based on the load balancing policy to obtain a query result.
The read-write separation data processing method based on Redis cache is realized based on Redis cluster, the Redis node maintains master _ key, and the dependency relationship of the master node and the slave node is decoupled; the data processing method of the embodiment of the application effectively reduces the configuration complexity of the Redis cluster, reduces the research, development, operation and maintenance cost, and can easily expand the application instance and the Redis node.
In some embodiments, the data writing step further comprises:
a write request acquisition step, configured to acquire a write request of the client and forward the write request to the application instance based on the load balancing policy;
a writing service step, which is used for the application instance to initiate a request to the database and write data;
and a write request hot spot caching step, which is used for acquiring the write-in key of the write-in request, processing hot spot data through a statistical module to obtain the hot spot key of the write-in key and circularly writing the hot spot key into the master _ key of each Redis node, wherein the master _ key does not set expiration time. And calculating and distributing the hotspot data through the statistical module.
In some embodiments, the data querying step further comprises:
a query request acquisition step, which is used for forwarding the query request of the client to the application instance based on the load balancing strategy;
a query request forwarding step, configured to forward, by using the Redis registry, the query request to a Redis node;
a Redis cache query step, which is used for obtaining a query result by sequentially reading the data in the master _ key;
a database query step, which is used for forwarding the query request to a database to obtain a query result when the query result cannot be obtained in the master _ key;
and a query request hot spot caching step, which is used for acquiring a query key of the query request entering the database query step, performing hot spot data processing on the query request by using the statistical module to obtain the hot spot key of the query key, and circularly writing the hot spot key of each Redis node.
In some embodiments, the data processing method further comprises: and a fusing processing step, which is used for directly forwarding the query request to a database for data query when the Redis cluster is unavailable.
In some embodiments, the statistical module performs hotspot data processing by using LinkedHashMap based on an LRU algorithm to realize distribution and update of hotspot keys and realize hotspot key calculation and elimination strategies. The LRU (Least recently used) algorithm eliminates data based on historical access records of the data, with the core idea being that "if the data was accessed recently, then the probability of future access is higher. And hot spot data can be actively added to the Redis node through a rule of a configuration statistical module.
In a second aspect, an embodiment of the present application provides a data processing system based on a Redis cache, including:
the Redis cluster building module is used for building a Redis cluster, configuring a load balancing strategy and configuring a Redis registration center;
the data writing module is used for acquiring a writing request of a client, writing the writing request of the client into a database through at least one application instance, and writing a writing key of the writing request into a master _ key of each Redis node in the Redis cluster after hot spot data processing is carried out on the writing key;
and the data query module is used for acquiring a query request of the client and forwarding the query request to the Redis node and/or the database to acquire a query result through the application instance based on the load balancing strategy.
The module realizes a read-write separation data processing method based on Redis cache based on a Redis cluster, and a Redis node maintains master _ key and decouples the dependency relationship of a master node and a slave node; the data processing method of the embodiment of the application effectively reduces the configuration complexity of the Redis cluster, reduces the research, development, operation and maintenance cost, and can easily expand the application instance and the Redis node.
In some embodiments, the data writing module further comprises:
a write request obtaining module, configured to obtain a write request of the client and forward the write request to the application instance based on the load balancing policy;
the writing service module is used for initiating a request to the database by the application example and writing data;
and the write request hot spot caching module is used for acquiring the write-in key of the write-in request, processing hot spot data through a statistical module to obtain the hot spot key of the write-in key and circularly writing the hot spot key into the master _ key of each Redis node, wherein the master _ key does not set expiration time.
In some embodiments, the data query module further comprises:
the query request acquisition module is used for forwarding a query request of a client to the application instance based on the load balancing strategy;
the query request forwarding module is used for forwarding the query request to a Redis node by utilizing the Redis registry;
the Redis cache query module is used for obtaining a query result by sequentially reading the data in the master _ key;
the database query module is used for forwarding the query request to a database to obtain a query result when the query result cannot be obtained in the master _ key;
and the query request hot spot cache module is used for acquiring a query key of the query request in the database query module, performing hot spot data processing on the query request by using the statistical module to obtain a hot spot key of the query key, and circularly writing the hot spot key of each Redis node.
In some of these embodiments, the data processing system further comprises:
and the fusing processing module is used for directly forwarding the query request to a database for data query when the Redis cluster is unavailable.
In some embodiments, the statistical module performs hotspot data processing by using LinkedHashMap based on LRU algorithm to realize distribution and update of hotspot keys. The LRU (Least recently used) algorithm eliminates data based on historical access records of the data, with the core idea being that "if the data was accessed recently, then the probability of future access is higher. According to the embodiment of the application, hot spot data can be actively added to the Redis node through a rule of a configuration statistical module.
In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor, when executing the computer program, implements the data processing method based on the Redis cache as described in the first aspect.
In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data processing method based on the Redis cache as described in the first aspect above.
Compared with the related art, the highly available, expandable and simplified data processing method, device, system, computer equipment and computer readable storage medium based on the Redis cache provided by the embodiment of the application reduce the complexity of the Redis cluster configuration and the Redis cluster application components, reduce the use complexity, and thus reduce the research and development and operation and maintenance costs; the read-write separation is realized, and the application instance and the Redis node can be easily expanded. The embodiment of the application also decouples the dependency relationship of the master node and the slave node.
The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a more thorough understanding of the application.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
FIG. 1 is a flow chart of a Redis cache-based data processing method according to an embodiment of the present application;
FIG. 2 is a flowchart illustrating substeps of step S2 of a Redis cache-based data processing method according to an embodiment of the present application;
FIG. 3 is a flowchart illustrating substeps of step S3 of a Redis cache-based data processing method according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a Redis cache-based data processing method according to an embodiment of the present application;
FIG. 5 is a flowchart illustrating a substep of step S2 of a Redis cache based data processing method according to a preferred embodiment of the present application;
FIG. 6 is a flowchart illustrating a substep of step S3 of a Redis cache based data processing method according to a preferred embodiment of the present application;
FIG. 7 is a block diagram of a Redis cache based data processing system according to an embodiment of the present application.
Description of the drawings:
1. a Redis cluster building module; 2. a data writing module; 3. a data query module;
201. a write request acquisition module; 202. writing into a service module;
203. a write request hotspot caching module; 301. a query request acquisition module;
302. a query request forwarding module; 303. a fusing processing module; 304. a Redis cache query module;
305. a database query module; 306. and the query request hot spot caching module.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments provided in the present application without any inventive step are within the scope of protection of the present application.
It is obvious that the drawings in the following description are only examples or embodiments of the present application, and that it is also possible for a person skilled in the art to apply the present application to other similar contexts on the basis of these drawings without inventive effort. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of ordinary skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments without conflict.
Unless defined otherwise, technical or scientific terms referred to herein shall have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar words throughout this application are not to be construed as limiting in number, and may refer to the singular or the plural. The present application is directed to the use of the terms "including," "comprising," "having," and any variations thereof, which are intended to cover non-exclusive inclusions; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or elements, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. Reference to "connected," "coupled," and the like in this application is not intended to be limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The term "plurality" as referred to herein means two or more. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. Reference herein to the terms "first," "second," "third," and the like, are merely to distinguish similar objects and do not denote a particular ordering for the objects.
The embodiment provides a data processing method based on Redis cache. Fig. 1 to 3 are flowcharts of a data processing method based on a Redis cache according to an embodiment of the present application, and fig. 4 is a schematic diagram of a data processing method based on a Redis cache according to a preferred embodiment of the present application, and as shown in fig. 1 to 3 and 4, the flowcharts include the following steps:
a Redis cluster building step S1, which is used for building a Redis cluster, configuring a load balancing strategy and configuring a Redis registry;
a data writing step S2, configured to obtain a write request of a client, write the write request of the client into a database through at least one application instance, perform hotspot data processing on a write key of the write request, and write the write key of the write request into a master _ key of each Redis node in the Redis cluster;
and a data query step S3, configured to obtain a query request of the client and forward the query request to the Redis node and/or the database through the application instance based on the load balancing policy to obtain a query result.
In some of these embodiments, the data writing step S2 further includes:
a write request obtaining step S201, configured to obtain a write request of a client and forward the write request to an application instance based on a load balancing policy;
a write service step S202, configured to initiate a request to a database by an application instance, and write data;
and a write request hot spot caching step S203, configured to obtain a write-in key of the write-in request, perform hot spot data processing through a statistical module to obtain the hot spot key of the write-in key, and write the hot spot key into the master _ key of each Redis node in a circulating manner, where the master _ key does not set an expiration time. And calculating and distributing the hotspot data through a statistical module.
In some of these embodiments, the data querying step S3 further includes:
a query request obtaining step S301, configured to forward a query request of a client to an application instance based on a load balancing policy;
a query request forwarding step S302, configured to forward a query request to a Redis node by using a Redis registry;
and a fusing processing step S303, configured to, when the Redis cluster is unavailable, directly forward the query request to the database for data query.
A Redis cache query step S304, configured to obtain a query result by sequentially reading data in the master _ key;
a database query step S305, configured to forward the query request to the database to obtain the query result when the master _ key cannot obtain the query result, and accordingly, if the master _ key directly queries the query result, this step may be skipped;
and a query request hot spot caching step S306, configured to obtain a query key of the query request entering the database query step, perform hot spot data processing on the query request by using the statistical module to obtain a hot spot key of the query key, and write the hot spot key of the query key into the master _ key of each Redis node in a circulating manner. Specifically, the statistical module performs hotspot data processing by using LinkedHashMap based on an LRU algorithm to realize distribution and update of hotspot keys and realize hotspot key calculation and elimination strategies. Optionally, in the embodiment of the present application, hot spot data may be actively added to the Redis node by configuring a rule of the statistics module.
Based on the steps, the data processing method based on read-write separation of Redis cache is realized based on the Redis cluster, the Redis node maintains master _ key, and the dependency relationship of the master node and the slave node is decoupled; the data processing method of the embodiment of the application effectively reduces the configuration complexity of the Redis cluster, reduces the research, development, operation and maintenance cost, and can easily expand the application instance and the Redis node.
The embodiments of the present application are described and illustrated below by means of preferred embodiments.
FIG. 5 is a flowchart illustrating a substep of step S2 of a Redis cache based data processing method according to a preferred embodiment of the present application; as shown in fig. 5, step S2 of the data processing method based on the Redis cache according to the embodiment of the present application includes the following steps:
step S501, forwarding a write-in request of a client to a specific application example based on a load balancing strategy;
step S502, the application example requests the database DB to write data according to the write request;
step S503, after the data writing is completed, the data corresponding to the writing request exists in the database DB, the written key is subjected to hot spot data processing by using LinkedHashMap, and is circularly written into the master _ key of each Redis node, and the hot spot data does not set expiration time.
Fig. 6 is a flowchart illustrating a substep of step S3 of the data processing method based on the Redis cache according to the preferred embodiment of the present application, and as shown in fig. 6, the data processing method based on the Redis cache includes the following steps:
step S601, forwarding the query request of the client to a specific application instance according to the configured load balancing strategy;
step S602, a Redis registry is used for accessing Redis nodes, and query requests are forwarded to the Redis nodes;
step S603, a Redis registry is used for accessing Redis to confirm whether the query request is successfully forwarded to the Redis node, if the query request is not successfully forwarded, the fusing processing condition is met, and the query request is directly forwarded to a database DB for data query;
step S604, if the query request is successfully forwarded, reading data on the master _ key of the Redis node according to the query condition of the query request, and returning a query result;
step S605, when the data on the master _ key is empty, the query request is forwarded to the database DB for query;
step S606, the query request of step S605 is circularly written into the master _ key of each Redis node after hot spot data analysis and calculation are carried out by the statistical module, and the hot spot data does not set expiration time.
It should be noted that the steps illustrated in the above-described flow diagrams or in the flow diagrams of the figures may be performed in a computer system, such as a set of computer-executable instructions, and that, although a logical order is illustrated in the flow diagrams, in some cases, the steps illustrated or described may be performed in an order different than here.
The present embodiment further provides a data processing system based on a Redis cache, where the system is used to implement the foregoing embodiments and preferred embodiments, and details are not repeated for what has been described. As used hereinafter, the terms "module," "unit," "subunit," and the like may implement a combination of software and/or hardware for a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 7 is a block diagram of a data processing system based on a Redis cache according to an embodiment of the present application, and referring to fig. 7, the system includes:
the Redis cluster building module 1 is used for building a Redis cluster, configuring a load balancing strategy and configuring a Redis registry;
the data writing module 2 is used for acquiring a writing request of a client, writing the writing request of the client into a database through at least one application instance, performing hot spot data processing on a writing key of the writing request, and writing the writing key of the writing request into a master _ key of each Redis node in the Redis cluster;
and the data query module 3 is configured to obtain a query request of a client and forward the query request to the Redis node and/or the database through an application instance based on a load balancing policy to obtain a query result.
Wherein, the data writing module 2 further comprises: a write request obtaining module 201, configured to obtain a write request of a client and forward the write request to an application instance based on a load balancing policy; the write-in service module 202 is used for initiating a request to the database by an application instance and writing data; the write request hot spot cache module 203 is configured to obtain a write-in key of the write-in request, perform hot spot data processing through a statistics module to obtain the hot spot key of the write-in key, and write the hot spot key into the master _ key of each Redis node in a circulating manner, where the master _ key does not set an expiration time.
Wherein, the data query module 3 further comprises: a query request obtaining module 301, configured to forward a query request of a client to an application instance based on a load balancing policy; a query request forwarding module 302, configured to forward a query request to a Redis node by using a Redis registry; the fusing processing module 303 is configured to, when the Redis cluster is unavailable, directly forward the query request to the database for data query; the Redis cache query module 304 is used for obtaining a query result by sequentially reading data in the master _ key; the database query module 305 is configured to forward the query request to the database to obtain the query result when the master _ key cannot obtain the query result, and accordingly, if the master _ key directly queries the query result, the module does not execute the query result. The query request hot spot cache module 306 is configured to obtain a query key of a query request in the database query module, perform hot spot data processing on the query request by using the statistics module to obtain a hot spot key of the query key, and write the hot spot key of the query key into the master _ key of each Redis node in a circulating manner. Specifically, the statistical module performs hotspot data processing by using LinkedHashMap based on an LRU algorithm to realize distribution and update of hotspot keys. Optionally, in the embodiment of the present application, hot spot data may be actively added to the Redis node through a rule of the configuration statistics module.
Based on the modules, the read-write separation data processing method based on Redis cache is realized based on the Redis cluster, the Redis node maintains master _ key, and the dependency relationship of the master node and the slave node is decoupled; the data processing method of the embodiment of the application effectively reduces the configuration complexity of the Redis cluster, reduces the research, development, operation and maintenance cost, and can easily expand the application instance and the Redis node.
The above modules may be functional modules or program modules, and may be implemented by software or hardware. For a module implemented by hardware, the modules may be located in the same processor; or the modules can be respectively positioned in different processors in any combination.
In addition, the data processing method based on the Redis cache according to the embodiment of the present application described in conjunction with FIG. 1 may be implemented by a computer device. The computer device may include a processor and a memory storing computer program instructions.
In particular, the processor may include a Central Processing Unit (CPU), or A Specific Integrated Circuit (ASIC), or may be configured to implement one or more Integrated circuits of the embodiments of the present Application.
The memory may include, among other things, mass storage for data or instructions. By way of example, and not limitation, memory may include a Hard Disk Drive (Hard Disk Drive, abbreviated to HDD), a floppy Disk Drive, a Solid State Drive (SSD), flash memory, an optical Disk, a magneto-optical Disk, tape, or a Universal Serial Bus (USB) Drive or a combination of two or more of these. The memory may include removable or non-removable (or fixed) media, where appropriate. The memory may be internal or external to the data processing apparatus, where appropriate. In a particular embodiment, the memory is a Non-Volatile (Non-Volatile) memory. In particular embodiments, the Memory includes Read-Only Memory (ROM) and Random Access Memory (RAM). The ROM may be mask-programmed ROM, Programmable ROM (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), Electrically rewritable ROM (EAROM), or FLASH Memory (FLASH), or a combination of two or more of these, where appropriate. The RAM may be a Static Random-Access Memory (SRAM) or a Dynamic Random-Access Memory (DRAM), where the DRAM may be a Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), an Extended data output Dynamic Random-Access Memory (EDODRAM), a Synchronous Dynamic Random-Access Memory (SDRAM), and the like.
The memory may be used to store or cache various data files for processing and/or communication use, as well as possibly computer program instructions for execution by the processor.
The processor reads and executes the computer program instructions stored in the memory to implement any one of the data processing methods based on the Redis cache in the above embodiments.
In addition, in combination with the data processing method based on the Redis cache in the foregoing embodiments, embodiments of the present application may provide a computer-readable storage medium to implement. The computer readable storage medium having stored thereon computer program instructions; the computer program instructions, when executed by a processor, implement any one of the Redis cache-based data processing methods in the above embodiments.
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A data processing method based on Redis cache is characterized by comprising the following steps:
a Redis cluster building step, which is used for building a Redis cluster, configuring a load balancing strategy and configuring a Redis registry;
a data writing step, configured to obtain a write request of a client, write the write request of the client into a database through at least one application instance, perform hotspot data processing on a write key of the write request, and write the write key of the write request into a master _ key of each Redis node in the Redis cluster;
and a data query step, configured to obtain a query request of the client and forward the query request to the Redis node and/or the database through the application instance based on the load balancing policy to obtain a query result.
2. The Redis cache-based data processing method according to claim 1, wherein the data writing step further comprises:
a write request acquisition step, configured to acquire a write request of the client and forward the write request to the application instance based on the load balancing policy;
a writing service step, which is used for the application instance to initiate a request to the database and write data;
and a write request hot spot caching step, configured to obtain a write-in key of the write-in request, perform hot spot data processing through a statistics module to obtain the hot spot key of the write-in key, and write the hot spot key of the write-in key into a master _ key of each Redis node in a circulating manner, where the master _ key does not set an expiration time.
3. The Redis cache-based data processing method according to claim 1, wherein the data querying step further comprises:
a query request acquisition step, which is used for forwarding the query request of the client to the application instance based on the load balancing strategy;
a query request forwarding step, configured to forward, by using the Redis registry, the query request to a Redis node;
a Redis cache query step, which is used for obtaining a query result by sequentially reading the data in the master _ key;
a database query step, which is used for forwarding the query request to a database to obtain a query result when the query result cannot be obtained in the master _ key;
and a query request hot spot caching step, which is used for acquiring a query key of the query request entering the database query step, performing hot spot data processing on the query request by using the statistical module to obtain the hot spot key of the query key, and circularly writing the hot spot key of each Redis node.
4. The Redis cache-based data processing method according to claim 1, further comprising:
and a fusing processing step, which is used for directly forwarding the query request to a database for data query when the Redis cluster is unavailable.
5. The Redis cache-based data processing method according to claim 2 or 3, wherein the statistical module performs hot spot data processing by using LinkedHashMap based on LRU algorithm.
6. A data processing system based on a Redis cache, comprising:
the Redis cluster building module is used for building a Redis cluster, configuring a load balancing strategy and configuring a Redis registration center;
the data writing module is used for acquiring a writing request of a client, writing the writing request of the client into a database through at least one application instance, and writing a writing key of the writing request into a master _ key of each Redis node in the Redis cluster after hot spot data processing is carried out on the writing key;
and the data query module is used for acquiring a query request of the client and forwarding the query request to the Redis node and/or the database to acquire a query result through the application instance based on the load balancing strategy.
7. The Redis cache-based data processing system of claim 6, wherein the data write module further comprises:
a write request obtaining module, configured to obtain a write request of the client and forward the write request to the application instance based on the load balancing policy;
the writing service module is used for initiating a request to the database by the application example and writing data;
and the write request hot spot caching module is used for acquiring the write-in key of the write-in request, processing hot spot data through a statistical module to obtain the hot spot key of the write-in key and circularly writing the hot spot key into the master _ key of each Redis node, wherein the master _ key does not set expiration time.
8. The Redis cache-based data processing system of claim 6, wherein the data query module further comprises:
the query request acquisition module is used for forwarding a query request of a client to the application instance based on the load balancing strategy;
the query request forwarding module is used for forwarding the query request to a Redis node by utilizing the Redis registry;
the Redis cache query module is used for obtaining a query result by sequentially reading the data in the master _ key;
the database query module is used for forwarding the query request to a database to obtain a query result when the query result cannot be obtained in the master _ key;
and the query request hot spot cache module is used for acquiring a query key of the query request in the database query module, performing hot spot data processing on the query request by using the statistical module to obtain a hot spot key of the query key, and circularly writing the hot spot key of each Redis node.
9. The Redis cache-based data processing system of claim 6, further comprising:
and the fusing processing module is used for directly forwarding the query request to a database for data query when the Redis cluster is unavailable.
10. Redis cache-based data processing system according to claim 7 or 8, wherein the statistical module utilizes LinkedHashMap for hot spot data processing based on LRU algorithm.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011564956.6A CN112579650A (en) | 2020-12-25 | 2020-12-25 | Data processing method and system based on Redis cache |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011564956.6A CN112579650A (en) | 2020-12-25 | 2020-12-25 | Data processing method and system based on Redis cache |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112579650A true CN112579650A (en) | 2021-03-30 |
Family
ID=75140608
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011564956.6A Pending CN112579650A (en) | 2020-12-25 | 2020-12-25 | Data processing method and system based on Redis cache |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112579650A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113377817A (en) * | 2021-06-28 | 2021-09-10 | 深圳市华曦达科技股份有限公司 | Data processing method, system, device and storage medium |
| CN117851456A (en) * | 2024-01-05 | 2024-04-09 | 迪爱斯信息技术股份有限公司 | Method, system and server for sharing data in cluster |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108183961A (en) * | 2018-01-04 | 2018-06-19 | 中电福富信息科技有限公司 | A kind of distributed caching method based on Redis |
| US20180210834A1 (en) * | 2017-01-20 | 2018-07-26 | Mz Ip Holdings, Llc | Systems and methods for reconstructing cache loss |
| CN108614847A (en) * | 2016-12-30 | 2018-10-02 | 北京京东尚科信息技术有限公司 | A kind of caching method and system of data |
| CN109753244A (en) * | 2018-12-29 | 2019-05-14 | 北京奥鹏远程教育中心有限公司 | A kind of application method of Redis cluster |
| CN110012098A (en) * | 2019-04-04 | 2019-07-12 | 浙江工业大学 | A kind of web high concurrent access process system and method |
| CN110933139A (en) * | 2019-11-05 | 2020-03-27 | 浙江工业大学 | System and method for solving high concurrency of Web server |
| CN111212145A (en) * | 2020-01-09 | 2020-05-29 | 国网福建省电力有限公司 | Redis cluster for power supply service command system |
| CN111324462A (en) * | 2020-02-20 | 2020-06-23 | 杭州梦视网络科技有限公司 | System and method with Web load balancing technology |
-
2020
- 2020-12-25 CN CN202011564956.6A patent/CN112579650A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108614847A (en) * | 2016-12-30 | 2018-10-02 | 北京京东尚科信息技术有限公司 | A kind of caching method and system of data |
| US20180210834A1 (en) * | 2017-01-20 | 2018-07-26 | Mz Ip Holdings, Llc | Systems and methods for reconstructing cache loss |
| CN108183961A (en) * | 2018-01-04 | 2018-06-19 | 中电福富信息科技有限公司 | A kind of distributed caching method based on Redis |
| CN109753244A (en) * | 2018-12-29 | 2019-05-14 | 北京奥鹏远程教育中心有限公司 | A kind of application method of Redis cluster |
| CN110012098A (en) * | 2019-04-04 | 2019-07-12 | 浙江工业大学 | A kind of web high concurrent access process system and method |
| CN110933139A (en) * | 2019-11-05 | 2020-03-27 | 浙江工业大学 | System and method for solving high concurrency of Web server |
| CN111212145A (en) * | 2020-01-09 | 2020-05-29 | 国网福建省电力有限公司 | Redis cluster for power supply service command system |
| CN111324462A (en) * | 2020-02-20 | 2020-06-23 | 杭州梦视网络科技有限公司 | System and method with Web load balancing technology |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113377817A (en) * | 2021-06-28 | 2021-09-10 | 深圳市华曦达科技股份有限公司 | Data processing method, system, device and storage medium |
| CN117851456A (en) * | 2024-01-05 | 2024-04-09 | 迪爱斯信息技术股份有限公司 | Method, system and server for sharing data in cluster |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20160112516A1 (en) | Distributed storage system, cluster node and range management method thereof | |
| US20190347167A1 (en) | Primary Node-Standby Node Data Transmission Method, Control Node, and Database System | |
| CN104935654A (en) | Caching method, write point client and read client in server cluster system | |
| CN104504147B (en) | A kind of resource coordination method of data-base cluster, apparatus and system | |
| CN108829720B (en) | Data processing method and device | |
| US20200349113A1 (en) | File storage method, deletion method, server and storage medium | |
| CN107241444B (en) | Distributed cache data management system, method and device | |
| US9213501B2 (en) | Efficient storage of small random changes to data on disk | |
| CN114443364B (en) | Distributed block storage data processing method, device, equipment and storage medium | |
| CN112579650A (en) | Data processing method and system based on Redis cache | |
| CN113010549A (en) | Data processing method based on remote multi-active system, related equipment and storage medium | |
| CN107943615B (en) | Data processing method and system based on distributed cluster | |
| CN112925828A (en) | Distributed multi-node query method and system | |
| CN114003662A (en) | Block chain performance optimization mechanism based on cache strategy | |
| CN111611223A (en) | Nonvolatile data access method, system, electronic device and medium | |
| CN114785662B (en) | Storage management method, device, equipment and machine-readable storage medium | |
| CN112130747A (en) | Distributed object storage system and data read-write method | |
| CN115794819A (en) | Data writing method and electronic equipment | |
| CN114428681A (en) | Method and device for expanding computing capacity of database system | |
| CN109558205B (en) | Disk access method and device | |
| CN117539915B (en) | Data processing method and related device | |
| CN113645261A (en) | Session data processing method, device, storage medium and computer equipment | |
| CN115374301B (en) | Cache device, method and system for realizing graph query based on cache device | |
| CN114579514B (en) | File processing method, device and equipment based on multiple computing nodes | |
| EP4394573A1 (en) | Data processing method and related device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |