CN118069740A - Method and system for synchronizing elastomer search data subscription type monitoring - Google Patents
Method and system for synchronizing elastomer search data subscription type monitoring Download PDFInfo
- Publication number
- CN118069740A CN118069740A CN202410198679.3A CN202410198679A CN118069740A CN 118069740 A CN118069740 A CN 118069740A CN 202410198679 A CN202410198679 A CN 202410198679A CN 118069740 A CN118069740 A CN 118069740A
- Authority
- CN
- China
- Prior art keywords
- data
- elastic search
- synchronization
- subscription
- processor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 77
- 238000012544 monitoring process Methods 0.000 title abstract description 9
- 229920001971 elastomer Polymers 0.000 title description 2
- 239000000806 elastomer Substances 0.000 title description 2
- 230000008859 change Effects 0.000 claims abstract description 20
- 238000004806 packaging method and process Methods 0.000 claims abstract description 8
- 230000001960 triggered effect Effects 0.000 claims abstract description 6
- 238000001914 filtration Methods 0.000 claims abstract description 3
- 230000001360 synchronised effect Effects 0.000 claims description 19
- 238000003860 storage Methods 0.000 claims description 15
- 238000000547 structure data Methods 0.000 claims description 9
- 238000013500 data storage Methods 0.000 claims description 3
- 238000003780 insertion Methods 0.000 claims description 3
- 230000037431 insertion Effects 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 238000001028 reflection method Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 abstract description 6
- 238000004590 computer program Methods 0.000 description 13
- 238000010586 diagram Methods 0.000 description 12
- 238000011161 development Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 238000011156 evaluation Methods 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000007547 defect Effects 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- HTRJZMPLPYYXIN-UHFFFAOYSA-N 2-acetamido-3-[[4-[(2-acetamido-2-carboxyethyl)sulfanylcarbothioylamino]phenyl]carbamothioylsulfanyl]propanoic acid Chemical compound CC(=O)NC(C(O)=O)CSC(=S)NC1=CC=C(NC(=S)SCC(NC(C)=O)C(O)=O)C=C1 HTRJZMPLPYYXIN-UHFFFAOYSA-N 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005538 encapsulation Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Landscapes
- Storage Device Security (AREA)
Abstract
The invention discloses an elastic search data subscription type monitoring synchronization method and system, wherein the method comprises the following steps: the method comprises the following steps: judging whether the business operation triggers the data operation or not according to the operation value transmitted by the front end; when triggered, executing data change; wherein the data operation comprises: at least one of adding, deleting and modifying; invoking ELASTICSEARCHAPI a method to trigger data engine data operation and obtaining operation change data; filtering the changed data, and converting and packaging the successfully filtered data; pushing the packaged data to a Kafka specified topic; judging whether an upper level exists; and synchronizing the packaged data to an upper center when the data is at the upper level. By the processing scheme, the synchronization of the elastic search can be realized rapidly, in real time and accurately.
Description
Technical Field
The invention relates to the technical field of computers, in particular to an elastic search data subscription type monitoring synchronization method and system.
Background
With the development of social informatization, the demands of enterprises, governments and institutions on the internet are increasing, and for some complex organizations, the enterprises, governments and institutions are required to deploy a subsystem for each subordinate or sub-company of the internet system product, the subsystems are not interfered with each other, and meanwhile, the headquarters are required to uniformly manage and control the subordinate, so that synchronous transmission of data of the subordinate and the subordinate is required, which means that high demands are made on real-time, high-efficiency and accuracy of data synchronization. For relational databases, there is a very mature solution, such as the tool for the synchronization problem of the threaded heterogeneous databases proposed by the ali. But this does not apply to all databases such as the elastiscearch.
Most of the database synchronization schemes in the market at present are to synchronize data through a synchronization tool, and the disadvantage is that the data in the elastic search is not updated timely and has poor real-time performance, or the synchronization of the data is performed through business operation, which is a small workload for developers.
The current common technical proposal is as follows:
The method has the defect that code service of data transmission is added to each service point operated by each data, the development workload of the method is large, and each service point is required to be ensured not to be missed.
Implementing synchronization through service customization development codes presents a problem that the sequency of data is difficult to guarantee
The data is synchronized by the synchronization tool, which generally uses a synchronization interval on the configuration tool to acquire the data at fixed time, and the problem that this method generally needs to manually configure a synchronized data structure and synchronization time, and the configured synchronization interval time can cause that the data cannot be synchronized in real time.
It is apparent that the above-mentioned existing synchronization method still has inconvenience and defects in use, and further improvement is needed. How to create a new synchronization method becomes the urgent need of improvement in the current industry.
Disclosure of Invention
In view of this, the embodiments of the present disclosure provide an elastic search data subscription type listening synchronization method, which at least partially solves the problems existing in the prior art.
In a first aspect, an embodiment of the present disclosure provides an elastic search data subscription type listening synchronization method, where the method includes the following steps:
Judging whether the business operation triggers the data operation or not according to the operation value transmitted by the front end; when triggered, executing data change; wherein the data operation comprises: at least one of adding, deleting and modifying;
Invoking ELASTICSEARCHAPI a method to trigger data engine data operation and obtaining operation change data;
Filtering the changed data, and converting and packaging the successfully filtered data;
pushing the packaged data to a Kafka specified topic;
Judging whether an upper level exists; and synchronizing the packaged data to the upper center when the upper level exists.
According to a specific implementation of an embodiment of the disclosure, the method further includes encryption based on data synchronization, including the steps of:
acquiring a center id to be synchronized;
Mixing id with the encapsulated data; carrying out exclusive OR operation on id and data:
ciphertext = original data XOR key
And carrying out md5 encryption on the confused data.
According to a specific implementation of an embodiment of the disclosure, the method further includes decryption based on data synchronization, including the steps of:
obtaining encrypted data to carry out md5 decryption;
performing data splitting based on sourceId of decrypted data to obtain the center id of the current level, and removing confusion ：
Original data = ciphertext XOR key;
and (5) obtaining the original data and warehousing.
According to a specific implementation manner of the embodiment of the disclosure, data pushed to a higher level by a lower level is received through a message subscription mode.
According to a specific implementation manner of the embodiment of the present disclosure, the calling ELASTICSEARCHAPI method triggers a data engine data operation, including:
When a data insertion request occurs, the elastic search returns the Hits of searchResponse;
Judging the operation type of the data by analyzing the hit number of the Hits and the returned Result value; the Result value includes: create, update, delete, not found, and not active;
the data is converted into a preset structure type by a reflection method according to the format type of the data, and the obtained structure data is packaged with one layer of data.
According to a specific implementation of an embodiment of the disclosure, the method further includes:
after the topic designated by Kafka receives the data, judging the operation type of the data, and mapping the corresponding structure data to the corresponding elastic search document according to an operation field in the data to carry out data storage.
In a second aspect, an embodiment of the present disclosure provides an elastic search data subscription type listening synchronization system, the system including:
The judging module is configured to judge whether the business operation triggers the data operation according to the operation value transmitted by the front end and the user; when triggered, executing data change; wherein the data operation comprises: at least one of adding, deleting and modifying;
The data acquisition module is configured to call ELASTICSEARCHAPI a method to trigger data operation of the data engine and acquire operation change data;
The packaging module is configured to filter the change data and convert and package the successfully filtered data;
A pushing module configured to push the encapsulated data onto a Kafka specified topic;
The synchronous module is configured to judge whether the upper level exists or not; and synchronizing the packaged data to the upper center when the upper level exists.
According to a specific implementation of an embodiment of the disclosure, the system further includes:
an encryption module configured to acquire a center id to be synchronized;
Mixing id with the encapsulated data; carrying out exclusive OR operation on id and data:
ciphertext = original data XOR key
And, carrying out md5 encryption on the confused data.
In a third aspect, embodiments of the present disclosure further provide an electronic device, including:
at least one processor; and
A memory communicatively coupled to the at least one processor; wherein,
The memory stores instructions executable by the at least one processor, which when executed by the at least one processor, cause the at least one processor to perform the method of the first aspect or any implementation of the first aspect.
In a fourth aspect, the presently disclosed embodiments also provide a non-transitory computer-readable storage medium storing computer instructions that, when executed by at least one processor, cause the at least one processor to perform the method of elastic search data subscription snoop synchronization in any implementation of the foregoing first aspect or first aspect.
In a fifth aspect, embodiments of the present disclosure also provide a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, cause the computer to perform the method of the first aspect or any implementation of the first aspect described above for elastic search data subscription listening synchronization.
The method for the subscription type monitoring synchronization of the elastic search data in the embodiment of the disclosure simplifies the synchronization development amount, optimizes the synchronization step, and can rapidly, real-timely and accurately realize the synchronization of the elastic search cascade data.
Drawings
The foregoing is merely an overview of the present invention, and the present invention is further described in detail below with reference to the accompanying drawings and detailed description.
Fig. 1 is a schematic flow chart of an elastic search data subscription type listening synchronization method provided in an embodiment of the present disclosure;
Fig. 2 is a schematic diagram of an elastic search data subscription type listening synchronization method according to an embodiment of the present disclosure;
Fig. 3 is a flowchart of an elastic search data subscription type listening synchronization method provided by an embodiment of the present disclosure;
FIG. 4 is a flowchart of a method for receiving data by Kafka according to an embodiment of the present disclosure;
FIG. 5 is a schematic diagram of a custom generic structure for synchronization of data according to an embodiment of the present disclosure;
Fig. 6 is a schematic diagram of a synchronous data encryption and decryption method according to an embodiment of the present disclosure;
Fig. 7 is a schematic structural diagram of an elastic search data subscription type listening synchronization system according to an embodiment of the present disclosure; and
Fig. 8 is a schematic diagram of an electronic device according to an embodiment of the disclosure.
Detailed Description
Embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.
Other advantages and effects of the present disclosure will become readily apparent to those skilled in the art from the following disclosure, which describes embodiments of the present disclosure by way of specific examples. It will be apparent that the described embodiments are merely some, but not all embodiments of the present disclosure. The disclosure may be embodied or practiced in other different specific embodiments, and details within the subject specification may be modified or changed from various points of view and applications without departing from the spirit of the disclosure. It should be noted that the following embodiments and features in the embodiments may be combined with each other without conflict. All other embodiments, which can be made by one of ordinary skill in the art without inventive effort, based on the embodiments in this disclosure are intended to be within the scope of this disclosure.
It is noted that various aspects of the embodiments are described below within the scope of the following claims. It should be apparent that the aspects described herein may be embodied in a wide variety of forms and that any specific structure and/or function described herein is merely illustrative. Based on the present disclosure, one skilled in the art will appreciate that one aspect described herein may be implemented independently of any other aspect, and that two or more of these aspects may be combined in various ways. For example, an apparatus may be implemented and/or a method practiced using any number of the aspects set forth herein. In addition, such apparatus may be implemented and/or such methods practiced using other structure and/or functionality in addition to one or more of the aspects set forth herein.
In addition, in the following description, specific details are provided in order to provide a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects may be practiced without these specific details.
According to the invention, a complete scheme is designed to realize the function of synchronizing the cascade data of the elastic search through less development workload, the request return result value of the API is obtained based on a SearchRequest class method of a Java platform by utilizing the particularity of the API call of the elastic search, corresponding modified data is obtained according to the result value and a corresponding unique document id, and then the transmission and consumption of the data are carried out by utilizing a Kafka message system.
The embodiment of the invention provides an elastic search data subscription type monitoring synchronization method, which realizes the effects of quick, light weight, convenience and universal data cascade synchronization. The principle of the invention is that the API request of the elastic search is monitored to return a value result, the obtained data is mapped into a corresponding structure in a table structure reflection mode, the obtained data is sent to a cascade platform of a child-parent stage through Kafka, and the child-parent stage carries out data change synchronization in a Kafka consumption mode. The key point of the invention is based on API return result monitoring, which ensures the accuracy of data, and ensures the consistency and the integrity of data by using the communication and consumption mechanism of Kafka.
The invention provides a solving method aiming at the problems in the prior art:
Starting from the Api request layer of ELASTICSEARSH, rather than from the service layer, the Api result code and bulkItem operation data are acquired when the Api requests for call, and the entity class and the interface in the elastic search corresponding to the entity change event are dynamically created through the acquired data, so that the data synchronization of the elastic search is realized, a user does not need to pay attention to the service point, directly calls the adding, deleting and modifying operation data of the elastic search, and the development quantity is reduced.
The data synchronization is performed through the elastic search request layer, the changing sequence of each piece of data is orderly, the changing sequence is performed according to the sequence of the service, on a data reporting channel, kafka (a high-throughput distributed publish-subscribe message system which can process all action stream data of a consumer in a website) is adopted, and the condition that a Kafka producer cannot lose production data can be ensured through configuring the message quality.
The invention is not realized by a synchronous tool, and adopts a data synchronous scheme which is made at a data processing layer, thereby having real-time triggering performance.
The invention does not operate on business codes and has decoupling property for codes on actual projects.
Elastic search: a distributed, high expansion, high real-time search and data analysis engine.
Kafka: kafka is an open source stream processing platform developed by the Apache software foundation.
Fig. 1 is a schematic diagram of a flow of an elastic search data subscription type listening synchronization method according to an embodiment of the present disclosure.
Fig. 2 is a diagram of an elastic search data subscription type listening synchronization method architecture corresponding to fig. 1.
Fig. 3 is a flow chart of an elastic search data subscription type listening synchronization method corresponding to fig. 1.
The whole scheme of the invention is divided into two parts, wherein one part is based on an API request of an elastic search to acquire synchronous data, and a set of isomorphic packaging method and processing method of the data, and the other part is based on a Kafka data communication consumption mode.
The elastic search can return searchResponse Hits of the Hits and returned results, and the results are classified into CREATED (created), UPDATED (updated), delete, NOT_FOUND (NOT FOUND), NOOP (NOT active) according to the Hits and returned results of the Hits, the corresponding data operation type is judged through analysis of the results, the corresponding data format type is used, the data is converted into the required structure type through reflection, the obtained structure data is packaged again, and the data is sent to the designated topic of the Kafka service of the other cascade center through the configured Kafka link. When the structural data is acquired, the aim of synchronizing the designated data can be achieved by configuring a monitoring mode, so that the transmission of redundant data can be prevented, and the load capacity of the data is reduced.
As shown in fig. 4, after Kafka receives the designated topic data, it determines the operation type of the data, maps the corresponding structure data to the corresponding elastic search document, and performs data warehouse entry. The table structures of the upper and lower centers are consistent, and the document_id defaults to a unique primary key identification.
The scheme mainly designs a set of universal data cascading model to achieve the synchronization of the majority of elastic search cascading data, so that additional synchronization codes are not needed to be added, and only main service codes are needed to be concerned.
As shown in fig. 5, the custom generic structure for synchronization of data includes: action, is used for identifying the operation type of the data; ADD new, UPDATE, DELETE; a table for recording a document name (data manipulation table name); data for storing the structure data (data structure body) of the operation, recording the data type and value; the time is used for recording the generation time of the data; sourceId are used to record the data source id, i.e., the center from which the data was generated.
As shown in fig. 1, at step S110, according to the operation value transmitted by the front end, it is determined whether the service operation triggers a data operation; when triggered, executing data change; wherein the data operation comprises: at least one of the addition, deletion and modification.
More specifically, step S120 is next followed.
At step S120, the call ELASTICSEARCHAPI method triggers the data engine data operation, obtaining operation change data.
More specifically, the call ELASTICSEARCHAPI method triggers the data engine data operation, fetching the operation hit data hits.
Next, the process goes to step S130.
At step S130, the change data is filtered, and the successfully filtered data is encapsulated.
In the embodiment of the present invention, the calling ELASTICSEARCHAPI method triggers the data engine data operation, including: when a data insertion request occurs, the elastic search returns the Hits of searchResponse; judging the operation type of the data by analyzing the hit number of the Hits and the returned Result value; the Result value includes: create, update, delete, not found, and not active; the data is converted into a preset structure type by a reflection method according to the format type of the data, and the obtained structure data is packaged with one layer of data.
More specifically, the hit data hits are traversed, hit results of result are acquired, which are CREATED (creation), UPDATED (update), delete, not_found, NOOP (no activity), and data of not_found and NOOP, respectively, are filtered out.
And matching the filtered data with a corresponding es structure body, obtaining a corresponding data type, constructing a data format, and packaging the data into json messages.
In an embodiment of the present invention, the method further includes encryption based on data synchronization, including the steps of: acquiring a center id to be synchronized; mixing id with the encapsulated data, performing exclusive OR operation on the id and the data: ciphertext = original data XOR key; and carrying out md5 encryption on the confused data.
In an embodiment of the present invention, the method further includes decryption based on data synchronization, including the steps of: obtaining encrypted data to carry out md5 decryption; performing data splitting based on sourceId of decrypted data to obtain the center id of the current level, and removing confusion ：
Original data = ciphertext XOR key;
and (5) obtaining the original data and warehousing.
More specifically, as shown in fig. 6, the encryption of the data synchronization includes the steps of:
step 1: and acquiring a center unique identifier which needs to be synchronized.
Step 2: the id is confused with the synchronization data.
And mixing the obtained center unique identification with the synchronous data, and splicing the data.
Step 3: and carrying out md5 encryption on the confused data.
Step 4: and obtaining the encrypted data to perform md5 decryption.
Step 5: and carrying out data splitting by taking sourceId of the synchronous data, and obtaining the original data by confusion removal for warehousing.
Next, the process goes to step S140.
At step S140, the encapsulated data is pushed onto a Kafka specified topic.
More specifically, the process is carried out,
In the embodiment of the invention, the data pushed by the lower level to the upper level is received in a message subscription mode.
In an embodiment of the present invention, the method further includes: after the topic designated by Kafka receives the data, judging the operation type of the data, and mapping the corresponding structure data to the corresponding elastic search document according to an operation field in the data to carry out data storage. ADD is new, MOD is modified, DELETE is deleted.
Next, the process goes to step S150.
At step S150, it is determined whether there is an upper stage; and synchronizing the packaged data to the upper center when the upper level exists.
The method for synchronizing the elastic search data subscription type monitoring simplifies the synchronization development quantity, optimizes the synchronization step, and can rapidly, real-timely and accurately realize the synchronization of the elastic search cascade data.
Fig. 7 shows an elastic search data subscription type listening synchronization system 700 provided by the present invention, which includes a judging module 710, a data obtaining module 720, a packaging module 730, a pushing module 740, and a synchronization module 750.
The judging module 710 is configured to judge whether the service operation triggers a data operation according to an operation value transmitted by the front end by the user; when triggered, executing data change; wherein the data operation comprises: at least one of adding, deleting and modifying;
The data acquisition module 720 is used for calling ELASTICSEARCHAPI a method to trigger data operation of the data engine and acquire operation change data;
the encapsulation module 730 is configured to filter the change data, and transform and encapsulate the filtered data;
the pushing module 740 is configured to push the encapsulated data onto a Kafka specified topic;
The synchronization module 750 is configured to determine whether there is an upper level; and synchronizing the packaged data to the upper center when the upper level exists.
In an embodiment of the present invention, the system further includes: an encryption module configured to acquire a center id to be synchronized; mixing id with the encapsulated data, performing exclusive OR operation on the id and the data: ciphertext = original data XOR key; and, carrying out md5 encryption on the confused data.
Referring to fig. 8, an embodiment of the present disclosure also provides an electronic device 80, including:
at least one processor; and
A memory communicatively coupled to the at least one processor; wherein,
The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the elastic search data subscription snoop synchronization method of the foregoing method embodiments.
The disclosed embodiments also provide a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the elastic search data subscription listening synchronization method in the foregoing method embodiments.
The disclosed embodiments also provide a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, cause the computer to perform the elastic search data subscription listening synchronization method of the foregoing method embodiments.
Referring now to fig. 8, a schematic diagram of an electronic device 80 suitable for use in implementing embodiments of the present disclosure is shown. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and the like, and stationary terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 8 is merely an example and should not be construed to limit the functionality and scope of use of the disclosed embodiments.
As shown in fig. 8, the electronic device 80 may include a processing means (e.g., a central processor, a graphics processor, etc.) 801 that may perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 802 or programs loaded from a storage 808 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 80 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other by a bus 804. An input/output (I/O) interface 805 is also connected to the bus 804.
In general, the following devices may be connected to the I/O interface 805: input devices 806 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; an output device 807 including, for example, a Liquid Crystal Display (LCD), speakers, vibrators, etc.; storage 808 including, for example, magnetic tape, hard disk, etc.; communication means 809. The communication means 809 may allow the electronic device 80 to communicate wirelessly or by wire with other devices to exchange data. While an electronic device 80 having various means is shown, it should be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via communication device 809, or installed from storage device 808, or installed from ROM 802. The above-described functions defined in the methods of the embodiments of the present disclosure are performed when the computer program is executed by the processing device 801.
It should be noted that the computer readable medium described in the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring at least two internet protocol addresses; sending a node evaluation request comprising the at least two internet protocol addresses to node evaluation equipment, wherein the node evaluation equipment selects an internet protocol address from the at least two internet protocol addresses and returns the internet protocol address; receiving an Internet protocol address returned by the node evaluation equipment; wherein the acquired internet protocol address indicates an edge node in the content distribution network.
Or the computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: receiving a node evaluation request comprising at least two internet protocol addresses; selecting an internet protocol address from the at least two internet protocol addresses; returning the selected internet protocol address; wherein the received internet protocol address indicates an edge node in the content distribution network.
Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units involved in the embodiments of the present disclosure may be implemented by means of software, or may be implemented by means of hardware. The name of the unit does not in any way constitute a limitation of the unit itself, for example the first acquisition unit may also be described as "unit acquiring at least two internet protocol addresses".
It should be understood that portions of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof.
The foregoing is merely specific embodiments of the disclosure, but the protection scope of the disclosure is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the disclosure are intended to be covered by the protection scope of the disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.
Claims (10)
1. An elastic search data subscription type listening synchronization method, which is characterized by comprising the following steps:
Judging whether the business operation triggers the data operation according to the operation value transmitted by the front end, and executing data change when the business operation triggers; wherein the data operation comprises: at least one of adding, deleting and modifying;
Invoking ELASTICSEARCHAPI a method to trigger data engine data operation and obtaining operation change data;
Filtering the changed data, and converting and packaging the successfully filtered data;
pushing the packaged data to a Kafka specified topic;
Judging whether an upper level exists; and synchronizing the packaged data to the upper center when the upper level exists.
2. The elastic search data subscription listening synchronization method of claim 1, wherein the method further comprises encryption based on data synchronization, comprising the steps of:
acquiring a center id to be synchronized;
mixing id with the encapsulated data, performing exclusive OR operation on the id and the data:
Ciphertext = original data XOR key;
and carrying out md5 encryption on the confused data.
3. The elastic search data subscription listening synchronization method of claim 2, wherein the method further comprises decryption based on data synchronization, comprising the steps of:
obtaining encrypted data to carry out md5 decryption;
performing data splitting based on sourceId of decrypted data to obtain the center id of the current level, and removing confusion ：
Original data = ciphertext XOR key;
and (5) obtaining the original data and warehousing.
4. The method for synchronizing the subscription-type listening of the elastic search data according to claim 1, wherein the data pushed from the lower level to the upper level is received through a message subscription manner.
5. The elastic search data subscription snoop synchronization method as claimed in claim 1, wherein said calling ELASTICSEARCHAPI method triggers a data engine data operation, comprising:
When a data insertion request occurs, the elastic search returns the Hits of searchResponse;
Judging the operation type of the data by analyzing the hit number of the Hits and the returned Result value; the Result value includes: create, update, delete, not found, and not active;
the data is converted into a preset structure type by a reflection method according to the format type of the data, and the obtained structure data is packaged with one layer of data.
6. The elastic search data subscription listening synchronization method of claim 1, wherein the method further comprises:
after the topic designated by Kafka receives the data, judging the operation type of the data, and mapping the corresponding structure data to the corresponding elastic search document according to an operation field in the data to carry out data storage.
7. An elastic search data subscription listening synchronization system, the system comprising:
The judging module is configured to judge whether the business operation triggers the data operation according to the operation value transmitted by the front end and the user; when triggered, executing data change; wherein the data operation comprises: at least one of adding, deleting and modifying;
The data acquisition module is configured to call ELASTICSEARCHAPI a method to trigger data operation of the data engine and acquire operation change data;
The packaging module is configured to filter the change data and convert and package the successfully filtered data;
A pushing module configured to push the encapsulated data onto a Kafka specified topic;
The synchronous module is configured to judge whether the upper level exists or not; and synchronizing the packaged data to the upper center when the upper level exists.
8. The elastic search data subscription listening synchronization system of claim 7, wherein the system further comprises:
an encryption module configured to acquire a center id to be synchronized;
mixing id with the encapsulated data, performing exclusive OR operation on the id and the data:
Ciphertext = original data XOR key;
and, carrying out md5 encryption on the confused data.
9. An electronic device, comprising:
at least one processor; and
A memory communicatively coupled to the at least one processor; wherein,
The memory stores instructions executable by the at least one processor, which when executed by the at least one processor, cause the at least one processor to perform the elastic search data subscription snoop synchronization method of any one of claims 1 to 6.
10. A non-transitory computer-readable storage medium storing computer instructions that, when executed by at least one processor, cause the at least one processor to perform the elastic search data subscription listening synchronization method of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410198679.3A CN118069740A (en) | 2024-02-22 | 2024-02-22 | Method and system for synchronizing elastomer search data subscription type monitoring |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410198679.3A CN118069740A (en) | 2024-02-22 | 2024-02-22 | Method and system for synchronizing elastomer search data subscription type monitoring |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118069740A true CN118069740A (en) | 2024-05-24 |
Family
ID=91096702
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410198679.3A Pending CN118069740A (en) | 2024-02-22 | 2024-02-22 | Method and system for synchronizing elastomer search data subscription type monitoring |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118069740A (en) |
-
2024
- 2024-02-22 CN CN202410198679.3A patent/CN118069740A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109582310B (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| CN111198859B (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| CN110704000B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN111274503B (en) | Data processing method, device, electronic equipment and computer readable medium | |
| CN111857720B (en) | User interface state information generation method and device, electronic equipment and medium | |
| CN112015816A (en) | Data synchronization method, device, medium and electronic equipment | |
| CN112035529A (en) | Caching method and device, electronic equipment and computer readable storage medium | |
| CN112199923A (en) | Identification generation method, system, device and medium based on distributed system | |
| CN111694629A (en) | Information display method and device and electronic equipment | |
| CN110545313B (en) | Message push control method and device and electronic equipment | |
| CN111274104B (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| CN111198853B (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| CN111177260A (en) | Database remote copying method and device and electronic equipment | |
| CN111290861A (en) | Message processing method and device and electronic equipment | |
| WO2023273564A1 (en) | Virtual machine memory management method and apparatus, storage medium, and electronic device | |
| CN113886353B (en) | Data configuration recommendation method and device for hierarchical storage management software and storage medium | |
| CN116108109A (en) | Cascade data synchronization method, system, equipment and storage medium based on Canal | |
| CN113420400B (en) | Routing relation establishment method, request processing method, device and equipment | |
| CN111625326B (en) | Task pipeline execution method and device and electronic equipment | |
| CN118069740A (en) | Method and system for synchronizing elastomer search data subscription type monitoring | |
| CN110727694B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN112799863B (en) | Method and device for outputting information | |
| CN111382057B (en) | Test case generation method, test method and device, server and storage medium | |
| CN111310175A (en) | iOS application safety monitoring and protecting method and device based on plug-in | |
| CN111552705A (en) | Data processing method and device based on chart, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |