CN111506587A - Remote double-living method based on Redis - Google Patents
Remote double-living method based on Redis Download PDFInfo
- Publication number
- CN111506587A CN111506587A CN202010242382.4A CN202010242382A CN111506587A CN 111506587 A CN111506587 A CN 111506587A CN 202010242382 A CN202010242382 A CN 202010242382A CN 111506587 A CN111506587 A CN 111506587A
- Authority
- CN
- China
- Prior art keywords
- data
- machine room
- message queue
- redis
- monitoring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 24
- 238000012544 monitoring process Methods 0.000 claims abstract description 37
- 238000012545 processing Methods 0.000 claims description 5
- 238000004519 manufacturing process Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 claims description 3
- 230000004044 response Effects 0.000 claims description 3
- 238000004891 communication Methods 0.000 abstract description 2
- 230000009545 invasion Effects 0.000 abstract description 2
- 230000001360 synchronised effect Effects 0.000 description 10
- 238000011161 development Methods 0.000 description 2
- 230000003139 buffering effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computer Security & Cryptography (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a remote double-active method based on Redis, which comprises a machine room I and a machine room II, wherein each machine room comprises a service layer, a database, a monitoring component, a message queue and a mirror image data component; the database is used for storing the data of the service layer of the machine room where the database is located and the data transmitted by the monitoring component; the monitoring component is used for monitoring and writing the updating data of the database and the message queue; the message queue is used for realizing the communication between the machine room monitoring component where the message queue is located and the mirror image data components of other machine rooms; the mirror image data assembly is used for making mirror image data between different machine rooms. The remote double-live method based on Redis provided by the invention has the advantages of low service invasion, high expandability and convenient implementation.
Description
Technical Field
The invention belongs to the field of computers, and particularly relates to a remote double-activity method based on Redis.
Background
Redis is completely open source and free, written in C language, complies with BSD protocol, is a high performance distributed memory database, runs on memory and supports a persisted NoSQ L database Redis provides Keyspace Notifications functionality since 2.8.0 later, allowing clients to subscribe to Pub/Sub channels to receive events affecting Redis datasets in some way.
With the rapid development of business, for many companies, the technical architecture built in a single region faces various problems such as the following: the limited infrastructure limits the scalability of the service; the machine room and city level fault disasters affect the sustainability of the service. The method adopted by the internet companies which are increasingly large in size in the last years is almost the inevitable choice for the development of large-scale application to a certain stage. The existing remote dual-active method has the advantages that the information of the message queue is directly synchronized through the DTS, and then the data distribution is carried out through the message queue, so that the method has great invasion to the service; the data of two places are synchronized through the reading assembly, each region needing to be synchronized is provided with a reading assembly with poor expandability, and the pressure is increased linearly along with the increase of the number of the synchronized regions; in addition, a dynamite cache system realized by adopting a Netflix open source writes in each Node needing synchronization after receiving data through an agent layer, Dyno Node nodes need to be deployed by the nodes, the data has redundancy synchronization, if the data does not have redundancy, the access side standards are not consistent, the operation and maintenance are not uniform, the read-write performance is larger than the native difference, and the difference is mainly reflected on the write, and the difference is more serious along with the increase of the number of the nodes.
Disclosure of Invention
In view of the above, the present invention provides a remote double-active method based on Redis to solve the problems of poor expandability and poor message synchronization consistency of remote double-active.
In order to achieve the purpose, the technical method of the invention is realized as follows:
a remote double-live method based on Redis comprises the following steps:
step 1: the service layers of the first machine room and the second machine room write data into the database of the machine room;
step 2: monitoring the data updating of the first database by a monitoring component of the first machine room, and writing the updated data into a message queue;
and step 3: the message queue of the first machine room writes the updated data into the message queue of the second machine room through the mirror image data assembly of the second machine room;
and 4, step 4: monitoring the message queue updating data of the machine room where the monitoring component of the machine room II is located, and writing the updating data into the database II;
and 5: and the second service pair carries out data acquisition on the second database to complete information synchronization.
Further, the databases used in step 1, step 2, step 4, and step 5 are redis databases.
Further, the message queue utilized in the step 2 to the step 4 is a kafka message queue, and the mirror image data component utilized in the step 3 is a MirrorMaker.
Further, in step 3, the message queue starts idempotent production for ensuring that the sender only sends the message once in the data synchronization process, and simultaneously, the response value for ensuring that the receiver only receives the message once is set to-1.
Furthermore, the monitoring service of the first machine room in the step 2 marks the updated data when monitoring the data update, and the monitoring component of the second machine room in the step 4 judges the mark after monitoring the data update, and does not perform synchronous processing on the received data.
Compared with the prior art, the invention has the following advantages:
(1) the information queue and the mirror image data component thereof are adopted to realize the synchronization of data, the service intrusiveness is low, and the expandability is high;
(2) and the data is acquired by adopting a redis database monitoring service, so that the consistency of the synchronous information is ensured.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate an embodiment of the invention and, together with the description, serve to explain the invention and not to limit the invention. In the drawings:
fig. 1 is a schematic diagram of a remote double-live method based on Redis according to an embodiment of the present invention.
Detailed Description
It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict.
In the description of the present invention, it is to be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", and the like, indicate orientations or positional relationships based on those shown in the drawings, and are used only for convenience in describing the present invention and for simplicity in description, and do not indicate or imply that the referenced devices or elements must have a particular orientation, be constructed and operated in a particular orientation, and thus, are not to be construed as limiting the present invention. Furthermore, the terms "first", "second", etc. are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first," "second," etc. may explicitly or implicitly include one or more of that feature. In the description of the present invention, "a plurality" means two or more unless otherwise specified.
In the description of the present invention, it should be noted that, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art through specific situations.
The present invention will be described in detail below with reference to the embodiments with reference to the attached drawings.
As shown in fig. 1, a remote double-active method based on Redis includes a first machine room and a second machine room, each machine room includes a service layer, a database, a monitoring component, a message queue, and a mirror data component;
the database is used for storing the data of the service layer of the machine room where the database is located and the data transmitted by the monitoring component;
the monitoring component is used for monitoring and writing the updating data of the database and the message queue, and mainly realizes the monitoring of updating, deleting, adding and expiration time and messages of Redis.
Redis' notification events are mainly key events: the key event notification takes keyevent @ as a prefix; all commands will only generate notifications after the key value has been altered, for example deleting foo will generate:
key event notification:
>>“pmessage”,"__key*__:*","__keyevent@0__:set",“foo”
the key is mainly concerned with the key, the value can be inquired through the key after the key is acquired, all data are further acquired to achieve synchronization, so that the key event with keyevent @ as the prefix is monitored, and then the service processing is carried out after the event is monitored.
The message queue is used for realizing communication between the monitoring component of the machine room where the message queue is located and the mirror image data component of other machine rooms, the method adopts kafka and a synchronization component MirrorMaker (mirror image data component) thereof, data monitored by Redis only needs to be written into corresponding synchronization topic (theme), and then the MirrorMaker can carry out data synchronization aiming at the topic synchronization strategy of multiple machine rooms;
the mirror image data assembly is used for making mirror image data between different machine rooms.
The database is a redis database.
The message queue adopts a kafka message queue.
As shown in fig. 1, the remote double-live method based on Redis includes the following processes:
the method comprises the following steps: the service layer of each machine room writes data into a database of the machine room;
step two: monitoring the data updating of the first database by a monitoring component of the first machine room, and writing the updated data into a message queue;
step three: the message queue of the first machine room synchronizes the updated data to the message queue of the second machine room through the mirror image data component of the second machine room;
step four: monitoring the message queue updating data of the machine room where the monitoring component of the machine room II is located, and writing the updating data into the database II;
step five: and the service pair carries out access to the second database to complete information synchronization.
The message queue opens enable of kafka, i.e. enables idempotent production in which a sender is guaranteed to send messages only once, and simultaneously sets a response value for guaranteeing that a receiver receives messages only once to be-1, i.e. ack setting-1.
The monitoring service of the first machine room marks the updated data when monitoring the data update, and the monitoring component of the second machine room judges the mark after monitoring the data update, and does not perform synchronous processing on the received data. For example, the source field is marked as the machine room from which the data is synchronized, if the data is found to be synchronized from the far end, even if the redis monitoring service acquires the key of the event, the key is discarded without performing synchronization processing, so that data buffering is realized, that is, the problem of data cycle replication caused by the fact that the monitoring service of the second machine room acquires the data again and synchronizes to the first machine room again after the data read from the first machine room redis is synchronized to the second machine room through the message queue MQ is solved.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (5)
1. A remote double-live method based on Redis is characterized by comprising the following steps:
step 1: the service layers of the first machine room and the second machine room write data into the database of the machine room;
step 2: monitoring the data updating of the first database by a monitoring component of the first machine room, and writing the updated data into a message queue;
and step 3: the message queue of the first machine room writes the updated data into the message queue of the second machine room through the mirror image data assembly of the second machine room;
and 4, step 4: monitoring the message queue updating data of the machine room where the monitoring component of the machine room II is located, and writing the updating data into the database II;
and 5: and the second service pair carries out data acquisition on the second database to complete information synchronization.
2. The Redis-based displaced double-live method according to claim 1, wherein: the databases utilized in the steps 1, 2, 4 and 5 are redis databases.
3. The Redis-based displaced double-live method according to claim 1, wherein: the message queue utilized in the steps 2 to 4 is kafka message queue, and the mirror data component utilized in the step 3 is MirrorMaker.
4. The Redis-based displaced double-live method according to claim 1, wherein: in step 3, the message queue starts idempotent production for ensuring that the sender only sends the message once in the data synchronization process, and simultaneously, the response value for ensuring that the receiver only receives the message once is set as-1.
5. The Redis-based displaced double-live method according to claim 1, wherein: and (4) marking the updated data when the monitoring service of the machine room I monitors the data updating, judging the marking of the monitoring component of the machine room II after monitoring the data updating, and not synchronously processing the received data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010242382.4A CN111506587A (en) | 2020-03-31 | 2020-03-31 | Remote double-living method based on Redis |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010242382.4A CN111506587A (en) | 2020-03-31 | 2020-03-31 | Remote double-living method based on Redis |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111506587A true CN111506587A (en) | 2020-08-07 |
Family
ID=71864745
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010242382.4A Pending CN111506587A (en) | 2020-03-31 | 2020-03-31 | Remote double-living method based on Redis |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111506587A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113726877A (en) * | 2021-08-26 | 2021-11-30 | 湖南快乐阳光互动娱乐传媒有限公司 | Method and system for storing data across machine room cluster |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1567238A (en) * | 2003-06-09 | 2005-01-19 | 四川大学 | Mirror method for files based on Internet |
| CN104202375A (en) * | 2014-08-22 | 2014-12-10 | 广州华多网络科技有限公司 | Method and system for synchronous data |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| CN108228397A (en) * | 2016-12-22 | 2018-06-29 | 深圳市优朋普乐传媒发展有限公司 | The method and apparatus that a kind of cluster span computer room synchronizes |
| CN109284312A (en) * | 2018-08-27 | 2019-01-29 | 山东威尔数据股份有限公司 | A kind of heterogeneous database change real-time informing method |
| CN110597910A (en) * | 2019-09-12 | 2019-12-20 | 聚好看科技股份有限公司 | Remote data synchronization method, device and system |
| CN110633320A (en) * | 2018-05-30 | 2019-12-31 | 北京京东尚科信息技术有限公司 | Processing method, system, equipment and storage medium of distributed data service |
-
2020
- 2020-03-31 CN CN202010242382.4A patent/CN111506587A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1567238A (en) * | 2003-06-09 | 2005-01-19 | 四川大学 | Mirror method for files based on Internet |
| CN104202375A (en) * | 2014-08-22 | 2014-12-10 | 广州华多网络科技有限公司 | Method and system for synchronous data |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| CN108228397A (en) * | 2016-12-22 | 2018-06-29 | 深圳市优朋普乐传媒发展有限公司 | The method and apparatus that a kind of cluster span computer room synchronizes |
| CN110633320A (en) * | 2018-05-30 | 2019-12-31 | 北京京东尚科信息技术有限公司 | Processing method, system, equipment and storage medium of distributed data service |
| CN109284312A (en) * | 2018-08-27 | 2019-01-29 | 山东威尔数据股份有限公司 | A kind of heterogeneous database change real-time informing method |
| CN110597910A (en) * | 2019-09-12 | 2019-12-20 | 聚好看科技股份有限公司 | Remote data synchronization method, device and system |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113726877A (en) * | 2021-08-26 | 2021-11-30 | 湖南快乐阳光互动娱乐传媒有限公司 | Method and system for storing data across machine room cluster |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111581284B (en) | Database high availability method, device, system and storage medium | |
| US20090144220A1 (en) | System for storing distributed hashtables | |
| CN110597910A (en) | Remote data synchronization method, device and system | |
| CN111143382A (en) | Data processing method, system and computer readable storage medium | |
| CN103701913A (en) | Data synchronization method and device | |
| CN108228581B (en) | Zookeeper compatible communication method, server and system | |
| CN103138912A (en) | Data synchronizing method and data synchronizing system | |
| CN112612851B (en) | Multi-center data synchronization method and device | |
| CN114979158B (en) | Resource monitoring method, system, equipment and computer readable storage medium | |
| CN113094430B (en) | Data processing method, device, equipment and storage medium | |
| CN112417042A (en) | Method and device for processing service request | |
| CN105787129B (en) | Data storage method and electronic equipment | |
| US20100274761A1 (en) | System and method for reliable symmetric data synchronization | |
| CN104580425A (en) | Client data synchronization method and system | |
| CN111506587A (en) | Remote double-living method based on Redis | |
| CN101841425B (en) | Network backup method, device and system without proxy | |
| WO2021212493A1 (en) | Data synchronization method and device, data storage system and computer-readable medium | |
| CN113076298A (en) | Distributed small file storage system | |
| CN107528703B (en) | Method and equipment for managing node equipment in distributed system | |
| CN116303789A (en) | Parallel synchronization method and device for multi-fragment multi-copy database and readable medium | |
| CN109257403B (en) | Data storage method and device and distributed storage system | |
| CN115964436A (en) | Method and device for managing database cluster | |
| CN114051036A (en) | Data synchronization method, device and equipment for rail transit signal system and storage medium | |
| CN114217986A (en) | Data processing method, device, equipment, storage medium and product | |
| CN112799835A (en) | Method and system for processing metadata of distributed database system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200807 |
|
| RJ01 | Rejection of invention patent application after publication |