CN111400350A - Configuration data reading method, system, electronic device and storage medium - Google Patents
Configuration data reading method, system, electronic device and storage medium Download PDFInfo
- Publication number
- CN111400350A CN111400350A CN202010177947.5A CN202010177947A CN111400350A CN 111400350 A CN111400350 A CN 111400350A CN 202010177947 A CN202010177947 A CN 202010177947A CN 111400350 A CN111400350 A CN 111400350A
- Authority
- CN
- China
- Prior art keywords
- configuration data
- reading
- cache
- server
- configuration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of data processing, and provides a configuration data reading method, a configuration data reading system, electronic equipment and a storage medium. The configuration data reading method comprises the following steps: the interface server responds to a configuration data reading request and reads configuration data from a local cache; if the configuration data matched with the configuration data reading request is read from the local cache, returning the configuration data read from the local cache; if the local cache does not have the configuration data matched with the configuration data reading request, reading the configuration data from the distributed cache, wherein the reading comprises the following steps: searching a readable cache server through a snapshot service configured in a distributed cache; reading configuration data matched with the configuration data reading request from the readable cache server; and returning the configuration data read from the readable cache server. The invention can realize the high-efficiency reading of the variable relational configuration data and meet the reading requirements of real-time property, consistency, high concurrency, high performance and high availability.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method and a system for reading configuration data, an electronic device, and a storage medium.
Background
The internet industry has a number of business needs for efficient reading of configuration data. For example, a reader of an a/B experiment needs to read configuration data of the a/B experiment in order to perform offloading of each version; the application program (APP) needs to read configuration data of different user preferences in order to perform personalized main interface presentation.
The service requirements are usually realized by adopting an SOA (service oriented architecture) interface and a Redis database, a reader calls the SOA interface, and the SOA interface directly reads configuration data in the Redis database and returns the configuration data to the reader. This design can basically guarantee real-time reading of configuration data, but two general disadvantages exist.
Firstly, the method cannot support efficient reading of relational configuration data, and independent configuration data in a Redis database needs to be read out and then correlated in an SOA interface. Therefore, the efficiency of the SOA interface is influenced, and when the business is changed, the SOA interface needs to recode the association relations and can take effect only after being released to production;
secondly, the Redis database can expand data storage nodes under the condition of data increase, but Hash marking the positions of the data storage nodes needs to be calculated again. Therefore, some configuration data cannot be accessed under the condition of capacity expansion, and in addition, under the condition of large configuration data increment, a relatively long time is needed for writing into the Redis database, so that different configuration data can be read by each node deploying the SOA at the same time, and data inconsistency is generated.
It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the invention and therefore may include information that does not constitute prior art that is already known to a person of ordinary skill in the art.
Disclosure of Invention
In view of this, the present invention provides a method, a system, an electronic device and a storage medium for reading configuration data, which can realize efficient reading of variable relational configuration data.
One aspect of the present invention provides a configuration data reading method, including the steps of: the interface server responds to a configuration data reading request and reads configuration data from a local cache; if the configuration data matched with the configuration data reading request is read from the local cache, returning the configuration data read from the local cache; if the local cache does not have configuration data matched with the configuration data reading request, the interface server reads the configuration data from the distributed cache, including: searching a readable cache server through a snapshot service configured in the distributed cache; reading the configuration data matched with the configuration data reading request from the readable cache server; and returning the configuration data read from the readable cache server.
In some embodiments, the configuration data reading method further comprises the step of periodically reading the full configuration data, the step of periodically reading the full configuration data comprising: the snapshot service periodically associates all configuration data in the database cluster and writes the associated configuration data into a configuration wide table of the database cluster; and the snapshot service reads the full configuration data from the configuration wide table and writes the full configuration data into the distributed cache.
In some embodiments, the step of writing to the distributed cache comprises: searching a writable cache server in the distributed cache; emptying the writable cache server, and writing the read total configuration data into the writable cache server; and after the writing is finished, marking the writable cache server as an updated readable cache server, and marking the readable cache server as the updated writable cache server.
In some embodiments, the step of periodically reading the full configuration data further comprises: and the interface server periodically reads the configuration data corresponding to the interface server from the readable cache server and updates the configuration data into the local cache.
In some embodiments, the method for reading configuration data further includes a step of reading incremental configuration data in real time, and the step of reading incremental configuration data in real time includes: when the database cluster receives the newly added configuration data, initiating a configuration change notification carrying the newly added configuration data; the distributed cache writes the newly added configuration data into a configuration wide table of the database cluster in a correlated manner through the snapshot service according to the configuration change notification; and the snapshot service reads incremental configuration data from the configuration wide table and writes the incremental configuration data into the distributed cache.
In some embodiments, the step of reading the incremental configuration data in real-time further comprises: when the distributed cache receives the increment configuration data, initiating a configuration change notification carrying the increment configuration data; and the interface server reads the incremental configuration data from the distributed cache and writes the incremental configuration data into the local cache according to the configuration change notification.
In some embodiments, in the step of reading the incremental configuration data in real time, the incremental configuration data is obtained in a long polling manner.
In some embodiments, the distributed cache includes a plurality of distributed cache servers, the snapshot service is configured on one cache server, and when the cache server configured with the snapshot service fails, the snapshot service is automatically started in real time on another cache server of the distributed cache.
In some embodiments, the interface server is configured in an interface service cluster, the interface service cluster includes a plurality of distributed interface servers, and each interface server responds to a configuration data read request corresponding to the interface server.
Another aspect of the present invention provides a configuration data reading system, including an interface server and a distributed cache, where the interface server includes: the reading request response module is used for responding to a configuration data reading request and reading configuration data from a local cache; a local cache reading module, configured to, when configuration data matching the configuration data reading request is read from the local cache, return the configuration data read from the local cache; and a distributed cache reading module, configured to read configuration data from the distributed cache when there is no configuration data matching the configuration data reading request in the local cache, including: searching a readable cache server through a snapshot service configured in the distributed cache; reading the configuration data matched with the configuration data reading request from the readable cache server; and returning the configuration data read from the readable cache server.
Yet another aspect of the present invention provides an electronic device including: a processor; a memory having stored therein executable instructions of the processor; wherein the processor is configured to perform the steps of the configuration data reading method of any of the embodiments described above via execution of the executable instructions.
Yet another aspect of the present invention provides a computer-readable storage medium storing a program that, when executed, implements the steps of the configuration data reading method described in any of the above embodiments.
Compared with the prior art, the invention has the beneficial effects that:
when receiving a configuration data reading request, the interface server searches configuration data from a local cache to improve the response speed and realize efficient reading of the configuration data;
the snapshot service ensures that the interface server reads the configuration data from the readable cache server when accessing the distributed cache, ensures the complete reading of the configuration data, and ensures the consistency of the configuration data.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
FIG. 1 is a schematic diagram illustrating an implementation architecture of a configuration data reading method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram illustrating steps of a configuration data reading method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram illustrating steps for periodically reading full configuration data according to an embodiment of the present invention;
FIG. 4 is a schematic diagram illustrating an implementation scenario of periodically reading full configuration data according to an embodiment of the present invention;
FIG. 5 is a schematic diagram illustrating steps for reading incremental configuration data in real time according to an embodiment of the present invention;
FIG. 6 is a schematic diagram illustrating an implementation scenario of reading incremental configuration data in real time according to an embodiment of the present invention;
FIG. 7 shows a block diagram of a configuration data reading system in an embodiment of the invention;
FIG. 8 is a schematic diagram showing a structure of an electronic apparatus according to an embodiment of the present invention; and
fig. 9 shows a schematic structural diagram of a computer-readable storage medium in an embodiment of the present invention.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar structures, and thus their repetitive description will be omitted.
Fig. 1 shows an implementation architecture of a configuration data reading method in an embodiment, and referring to fig. 1, the implementation architecture for implementing the configuration data reading method includes an interface Service layer 11 providing an interface Service, a distributed Cache 12 providing a distributed Cache Service, and a database cluster 13 storing configuration data, where the interface Service layer 11 has a plurality of interface Service clusters 110 disposed therein, each interface Service cluster 110 includes a plurality of distributed interface servers 111, in the embodiment, the interface servers 111 are constructed based on an SOA, and the SOA (Service-oriented architecture) uses an interface for data interaction, and has advantages of agility, consistency, flexibility, and the like, the interface Service of the interface Service layer 11 is distributed, and has available functions such as fusing, degrading, remoting, and auto capacity expansion, and the like, the distributed Cache 12 has a plurality of distributed Cache clusters 120 disposed therein, each distributed Cache cluster 120 includes a plurality of Cache servers 122, and the like, in the embodiment, the Cache servers 122 are Cache memories, and the layer of the distributed Cache 12 is located in the lower layer of the interface Service layer 11, and the distributed Cache 12 has data consistency, activity query SQ L, database cluster 13, and the database cluster 130 includes a database cluster 13.
In a specific implementation scenario, the implementation framework for implementing the configuration data reading method may be deployed on a background service of an a/B experiment splitter, and is used for receiving an a/B experiment splitting request and efficiently reading configuration data.
Fig. 2 shows the main steps of the configuration data reading method in the embodiment, and the configuration data reading method in this embodiment may be implemented by the implementation architecture shown in fig. 1. Referring to fig. 2, the method for reading configuration data in this embodiment mainly includes: in step S220, the interface server reads the configuration data from the local cache in response to the configuration data read request; in step S230, if the configuration data matching the configuration data reading request is read from the local cache, returning the configuration data read from the local cache; in step S240, if there is no configuration data matching the configuration data reading request in the local cache, the interface server reads the configuration data from the distributed cache, which specifically includes: step S2402, searching a readable cache server through a snapshot service configured in the distributed cache; step S2404, reading configuration data matched with the configuration data reading request from the readable cache server; and step S2406, returning the configuration data read from the readable cache server.
The configuration data is usually non-massive data with a small number of writes and a large number of reads, and the configuration data that the data reader wants to read is fixed and unchanged no matter how the relationship of the configuration data changes. For example, the configuration data of the a/B experiment, no matter how the split ratio, position, index, etc. of the experiment change, the data reader of the a/B experiment only needs to know the version, i.e. the experiment group, to which the a/B experiment number and the user ID are split. Therefore, the SOA interface for the configuration data can be fixed, and generally does not need to be changed after being brought online, so that each interface server can respond to the configuration data reading request corresponding to the interface server.
The method comprises the steps that when an interface server receives a configuration data reading request, configuration data are searched from a local cache (L ocalCache) to improve the response speed and achieve efficient reading of the configuration data, the local cache has certain storage capacity limitation and can store relatively active configuration data according to certain rules to improve the reading speed of common configuration data, according to the memory use condition of the interface server, all the configuration data can be stored in the local cache if the configuration data are not much, a COW (Copy-On-Write) mechanism can be adopted by the local cache of the interface server to effectively deal with the configuration data access requests with less writing and more reading, if the local cache does not have matched configuration data, the configuration data are read from the distributed cache, and a snapshot service deployed in the distributed cache ensures that the configuration data are read from a readable cache server when the interface server accesses the distributed cache, the complete reading of the configuration data is ensured, and the consistency of the configuration data is ensured.
The consistency requirement of the initiating client of the configuration data reading request on the configuration data is high, and the situation that different clients read different configuration data at the same time cannot occur, so that the consistency design is carried out on the interface server and the distributed cache through the snapshot service to meet the consistency reading requirement of the configuration data. The interface server reads the configuration data from the lower layer periodically in a pull mode, and the lower layer informs the upper layer in a push mode when the configuration data is changed so that the upper layer can read the configuration data in real time.
Fig. 3 shows the main steps of periodically reading the full configuration data in the embodiment, and referring to fig. 3, the configuration data reading method further includes a step S300 of periodically reading the full configuration data, where the process of periodically reading the full configuration data specifically includes: step S320, the snapshot service periodically associates all the configuration data in the database cluster, and writes the associated configuration data into a configuration wide table of the database cluster; and step S340, the snapshot service reads the full configuration data from the configuration wide table and writes the full configuration data into the distributed cache. Wherein the step of writing to the distributed cache comprises: step S3402, searching a writable cache server in the distributed cache; step S3404, emptying the writable cache server and writing the read total configuration data into the writable cache server; and step S3406, after the writing is finished, marking the writable cache server as an updated readable cache server, and marking the readable cache server as the updated writable cache server.
Specifically, referring to the implementation scenario of periodically reading the full configuration data shown in fig. 4, the configuration data is read in a pull mode. The snapshot service 121 deployed in the platform layer of the distributed cache 12 associates all the configuration data, for example, every 5 minutes, and writes the configuration data into the configuration wide table of the database cluster 13, where the configuration data and the association relationship of the configuration data are stored. The snapshot service 121 then reads the full configuration data from the configuration wide table and writes to the distributed cache 12. The snapshot service 121 will first empty the writable cache server 122a in the distributed cache 12, then write the full configuration data of the configuration wide table, and after the write is completed, mark the current writable cache server 122a as the updated readable cache server, and mark the current readable cache server 122b as the updated writable cache server. This allows the latest configuration data to be read from the updated readable cache server the next time an interface server 111a accesses the distributed cache 12. In the process of writing to the writable cache server 122a, the interface server 111a accesses the current readable cache server 122b, and the situation that the read configuration data is inconsistent due to writing is avoided. The snapshot service 121 may allow the batch configuration data to be used by the interface servers under the condition of completing association and writing, and ensure that each interface server reads the consistent configuration data at the same time.
Further, referring to fig. 3, the step S300 of periodically reading the full configuration data further includes: step S360, the interface server periodically reads the configuration data of the corresponding interface server from the readable cache server, and updates the configuration data to the local cache. The periodic reading of the configuration data by the interface server and the periodic reading of the data by the distributed cache may be performed simultaneously or separately, depending on the respective cycle times. In one embodiment, referring to fig. 4, the interface server 111a actively pulls the configuration data from the distributed cache 12 to the local cache 1110 every 5 minutes from the beginning of the whole point, so that even if the interface server 111a is pulled to pull the interface service cluster, the interface server 111a reads the configuration data from the distributed cache 12 at the same time as the other interface servers at a time point, and the consistency of the configuration data is guaranteed.
In the above embodiment of periodically reading the full configuration data, the design of periodically reading the configuration data is to prevent the impact on the downstream data, such as buffer penetration, buffer breakdown, buffer avalanche, etc.; and the configuration data itself is not changed much, so real-time full reading is not needed. In addition, if the configuration data in the distributed cache is not backed up to the permanent storage device, the condition of complete service failure can be caused when the local cache of the interface server fails, so the configuration wide table also has the function of enabling the interface server to have a record for recovering the data from the configuration wide table of the database cluster so as to recover the service.
Fig. 5 shows the main steps of reading the incremental configuration data in real time in the embodiment, and referring to fig. 5, the configuration data reading method further includes a step S500 of reading the incremental configuration data in real time, where the process of reading the incremental configuration data in real time specifically includes: step S520, when the database cluster receives the newly added configuration data, a configuration change notification carrying the newly added configuration data is initiated; step S530, the distributed cache writes the newly added configuration data into a configuration broad table of the database cluster through the snapshot service according to the configuration change notice; and step S540, the snapshot service reads the incremental configuration data from the configuration wide table and writes the incremental configuration data into the distributed cache. Further, the process of reading the incremental configuration data in real time further includes: step S550, when the distributed cache receives the increment configuration data, initiating a configuration change notification carrying the increment configuration data; and step S560, the interface server reads the incremental configuration data from the distributed cache and writes the incremental configuration data into the local cache according to the configuration change notification.
In an embodiment, the step S500 of reading the incremental configuration data in real time acquires the incremental configuration data in a long Polling (L ong-Polling) manner, where the long Polling is implemented by a program coordination service deployed among the interface service layer, the distributed cache, and the database cluster, and the program coordination service is, for example, a Zookeeper cluster, and can implement notification of the change message in real time.
Specifically, referring to the implementation scenario of reading incremental configuration data in real time shown in fig. 6, a process of reading changed configuration data in real time in a push mode is performed, after a long Polling L ong-Polling request is initiated by the interface server 111a, a response of the distributed cache 12 is waited, after the configuration data in the distributed cache 12 changes, a configuration change notification is sent to the Zookeeper cluster 14, wherein a relevant change flag of the incremental configuration data is carried, the Zookeeper cluster 14 generates a distributed cache data change item accordingly, so as to respond to a long Polling L ong-Polling request of the interface server 111a, after the interface server 111a can pull the incremental configuration data from the distributed cache 12 in real time according to the configuration change notification, and write the incremental configuration data into a local cache, similarly, after the distributed cache 12 initiates a long Polling L ong-Polling request, a response of the distributed cache cluster 13 is waited, after the configuration data in the database cluster 13 changes, a configuration change notification is sent to the Zookeeper cluster 14, wherein a relevant change flag of the configuration data is carried, a configuration change notification is generated, so as to refresh the configuration data in the Zookeeper cluster 14, and the configuration data in a snapshot table of the distributed database 12 is updated, and then the configuration data can be sent to the snapshot update the configuration change notification.
In the embodiment for reading the incremental configuration data, an L ong-Polling mechanism rather than a message notification mechanism is adopted to solve the problem of inconsistent data of each node of an interface server and a distributed cache platform, and simultaneously avoid the problem of excessive network long connection number.
In a specific application scenario, the configuration data reading method is deployed on an a/B experiment shunt background service, and receives each a/B experiment shunt request, the QPS (query rate per second) peak can reach 5000, and as the a/B experiments are continuously accessed, the QPS can also be continuously increased, so that an average QPS2000 is achieved, and a P99 (average delay of slowest 1% request in the past 10 seconds) line is maintained below 10ms, so that a good high-performance real-time reading effect of variable relational configuration data is achieved.
In summary, the configuration data reading method of the present invention searches the configuration data from the local cache first when receiving the configuration data reading request through the interface server, so as to improve the response speed and realize the efficient reading of the configuration data; the method comprises the steps that when an interface server accesses a distributed cache, configuration data are read from a readable cache server through a snapshot service, the complete reading of the configuration data is guaranteed, and the consistency of the configuration data is guaranteed; by regularly reading the full configuration data and reading the incremental configuration data in real time, the variable relational configuration data can be efficiently read, and the reading requirements of real-time performance, consistency, high concurrency, high performance and high availability are met.
Referring to fig. 7, the configuration data reading system includes an interface server 111 and a distributed cache 12, where the interface server 111 includes: a read request response module 1112, configured to respond to the configuration data read request, and read the configuration data from the local cache; a local cache reading module 1113, configured to, when configuration data matching the configuration data reading request is read from the local cache, return the configuration data read from the local cache; and a distributed cache reading module 1114, configured to read the configuration data from the distributed cache 12 when there is no configuration data in the local cache that matches the configuration data read request, including: looking up a readable cache server through a snapshot service configured in the distributed cache 12; reading configuration data matched with the configuration data reading request from the readable cache server; and returning the configuration data read from the readable cache server.
In some embodiments, the configuration data reading system may further include a functional module for implementing each step in any of the above-described configuration data reading method embodiments. The configuration data reading system searches the configuration data from the local cache when receiving the configuration data reading request through the interface server, improves the response speed and realizes the high-efficiency reading of the configuration data; the method comprises the steps that when an interface server accesses a distributed cache, configuration data are read from a readable cache server through a snapshot service, the complete reading of the configuration data is guaranteed, and the consistency of the configuration data is guaranteed; and the efficient reading of the variable relational configuration data can be realized by regularly reading the full configuration data and reading the incremental configuration data in real time, and the reading requirements of real-time performance, consistency, high concurrency, high performance and high availability are met.
The embodiment of the present invention further provides an electronic device, which includes a processor and a memory, where the memory stores executable instructions, and the processor is configured to execute the steps of the configuration data reading method in the foregoing embodiment by executing the executable instructions.
As described above, the electronic device of the present invention can search the configuration data from the local cache first when receiving the configuration data reading request through the interface server, thereby increasing the response speed and realizing efficient reading of the configuration data; the method comprises the steps that when an interface server accesses a distributed cache, configuration data are read from a readable cache server through a snapshot service, the complete reading of the configuration data is guaranteed, and the consistency of the configuration data is guaranteed; and the efficient reading of the variable relational configuration data can be realized by regularly reading the full configuration data and reading the incremental configuration data in real time, and the reading requirements of real-time performance, consistency, high concurrency, high performance and high availability are met.
Fig. 8 is a schematic structural diagram of an electronic device in an embodiment of the present invention, and it should be understood that fig. 8 only schematically illustrates various modules, and these modules may be virtual software modules or actual hardware modules, and the combination, the splitting, and the addition of the remaining modules of these modules are within the scope of the present invention.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" platform.
The electronic device 600 of the present invention is described below with reference to fig. 8. The electronic device 600 shown in fig. 8 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 8, the electronic device 600 is embodied in the form of a general purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one memory unit 620, a bus 630 connecting the different platform components (including the memory unit 620 and the processing unit 610), a display unit 640, etc.
Wherein the storage unit stores a program code, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps of the configuration data reading method described in the above embodiments. For example, processing unit 610 may perform the steps shown in fig. 2, 3, and 5.
The storage unit 620 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)6201 and/or a cache memory unit 6202, and may further include a read-only memory unit (ROM) 6203.
The memory unit 620 may also include programs/utilities 6204 including one or more program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The embodiment of the present invention further provides a computer-readable storage medium, which is used for storing a program, and when the program is executed, the steps of the configuration data reading method described in the above embodiment are implemented. In some possible embodiments, the various aspects of the invention may also be implemented in the form of a program product comprising program code means for causing a terminal device to carry out the steps of the configuration data reading method described in the above embodiments, when the program product is run on the terminal device.
As described above, the computer-readable storage medium of the present invention can search the configuration data from the local cache first when receiving the configuration data reading request through the interface server, thereby increasing the response speed and realizing efficient reading of the configuration data; the method comprises the steps that when an interface server accesses a distributed cache, configuration data are read from a readable cache server through a snapshot service, the complete reading of the configuration data is guaranteed, and the consistency of the configuration data is guaranteed; and the efficient reading of the variable relational configuration data can be realized by regularly reading the full configuration data and reading the incremental configuration data in real time, and the reading requirements of real-time performance, consistency, high concurrency, high performance and high availability are met.
Fig. 9 is a schematic structural diagram of a computer-readable storage medium of the present invention. Referring to fig. 9, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of readable storage media include, but are not limited to: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including AN object oriented programming language such as Java, C + +, or the like, as well as conventional procedural programming languages, such as the "C" language or similar programming languages.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.
Claims (12)
1. A method for reading configuration data is characterized by comprising the following steps:
the interface server responds to a configuration data reading request and reads configuration data from a local cache;
if the configuration data matched with the configuration data reading request is read from the local cache, returning the configuration data read from the local cache;
if the local cache does not have configuration data matched with the configuration data reading request, the interface server reads the configuration data from the distributed cache, including:
searching a readable cache server through a snapshot service configured in the distributed cache;
reading the configuration data matched with the configuration data reading request from the readable cache server; and
returning the configuration data read from the readable cache server.
2. The method of reading configuration data according to claim 1, further comprising the step of periodically reading full configuration data, the step of periodically reading full configuration data comprising:
the snapshot service periodically associates all configuration data in the database cluster and writes the associated configuration data into a configuration wide table of the database cluster; and
and the snapshot service reads the full configuration data from the configuration wide table and writes the full configuration data into the distributed cache.
3. The configuration data reading method of claim 2, wherein the step of writing to the distributed cache comprises:
searching a writable cache server in the distributed cache;
emptying the writable cache server, and writing the read total configuration data into the writable cache server; and
and after the writing is finished, marking the writable cache server as an updated readable cache server, and marking the readable cache server as the updated writable cache server.
4. The method for reading configuration data according to claim 2, wherein the step of periodically reading the full amount of configuration data further comprises:
and the interface server periodically reads the configuration data corresponding to the interface server from the readable cache server and updates the configuration data into the local cache.
5. The method for reading configuration data according to claim 1, further comprising the step of reading incremental configuration data in real time, the step of reading incremental configuration data in real time comprising:
when the database cluster receives the newly added configuration data, initiating a configuration change notification carrying the newly added configuration data;
the distributed cache writes the newly added configuration data into a configuration wide table of the database cluster in a correlated manner through the snapshot service according to the configuration change notification; and
and the snapshot service reads incremental configuration data from the configuration wide table and writes the incremental configuration data into the distributed cache.
6. The method for reading configuration data according to claim 5, wherein the step of reading incremental configuration data in real time further comprises:
when the distributed cache receives the increment configuration data, initiating a configuration change notification carrying the increment configuration data; and
and the interface server reads the incremental configuration data from the distributed cache and writes the incremental configuration data into the local cache according to the configuration change notification.
7. The method for reading configuration data according to claim 5 or 6, wherein in the step of reading incremental configuration data in real time, the incremental configuration data is obtained by long polling.
8. The method according to claim 1, wherein the distributed cache includes a plurality of distributed cache servers, the snapshot service is configured on one cache server, and when a cache server configuring the snapshot service fails, the snapshot service is automatically started in real time on another cache server of the distributed cache.
9. The method according to claim 1, wherein the interface server is configured in an interface service cluster, the interface service cluster includes a plurality of distributed interface servers, and each interface server responds to a configuration data reading request corresponding to the interface server.
10. A configuration data reading system comprising an interface server and a distributed cache, the interface server comprising:
the reading request response module is used for responding to a configuration data reading request and reading configuration data from a local cache;
a local cache reading module, configured to, when configuration data matching the configuration data reading request is read from the local cache, return the configuration data read from the local cache; and
a distributed cache reading module, configured to read configuration data from a distributed cache when there is no configuration data matching the configuration data reading request in the local cache, including: searching a readable cache server through a snapshot service configured in the distributed cache; reading the configuration data matched with the configuration data reading request from the readable cache server; and returning the configuration data read from the readable cache server.
11. An electronic device, comprising:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the configuration data reading method of any of claims 1 to 9 via execution of the executable instructions.
12. A computer-readable storage medium storing a program, wherein the program when executed implements the steps of the configuration data reading method according to any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010177947.5A CN111400350B (en) | 2020-03-13 | 2020-03-13 | Configuration data reading method, system, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010177947.5A CN111400350B (en) | 2020-03-13 | 2020-03-13 | Configuration data reading method, system, electronic device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111400350A true CN111400350A (en) | 2020-07-10 |
| CN111400350B CN111400350B (en) | 2023-05-02 |
Family
ID=71432506
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010177947.5A Active CN111400350B (en) | 2020-03-13 | 2020-03-13 | Configuration data reading method, system, electronic device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111400350B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112333036A (en) * | 2021-01-06 | 2021-02-05 | 广州技象科技有限公司 | Multi-storage-node-based power Internet of things configuration data backup method and device |
| CN112487028A (en) * | 2020-11-09 | 2021-03-12 | 杭州电魂网络科技股份有限公司 | Method, system, electronic device and storage medium for server data storage |
| CN115269660A (en) * | 2022-09-26 | 2022-11-01 | 平安银行股份有限公司 | Cache data processing method and device, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110271067A1 (en) * | 2010-05-03 | 2011-11-03 | Pixel8 Networks, Inc. | Efficient Cloud Network Attached Storage |
| CN102843396A (en) * | 2011-06-22 | 2012-12-26 | 中兴通讯股份有限公司 | Data writing and reading method and device in distributed caching system |
| CN104052824A (en) * | 2014-07-04 | 2014-09-17 | 哈尔滨工业大学深圳研究生院 | Distributed cache method and system |
| US8880812B1 (en) * | 2010-03-31 | 2014-11-04 | Marvell International Ltd. | WWN table management systems and methods |
| CN107071059A (en) * | 2017-05-25 | 2017-08-18 | 腾讯科技（深圳）有限公司 | Distributed caching service implementing method, device, terminal, server and system |
| CN110555041A (en) * | 2018-03-30 | 2019-12-10 | 腾讯科技（深圳）有限公司 | Data processing method, data processing device, computer equipment and storage medium |
-
2020
- 2020-03-13 CN CN202010177947.5A patent/CN111400350B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8880812B1 (en) * | 2010-03-31 | 2014-11-04 | Marvell International Ltd. | WWN table management systems and methods |
| US20110271067A1 (en) * | 2010-05-03 | 2011-11-03 | Pixel8 Networks, Inc. | Efficient Cloud Network Attached Storage |
| CN102843396A (en) * | 2011-06-22 | 2012-12-26 | 中兴通讯股份有限公司 | Data writing and reading method and device in distributed caching system |
| CN104052824A (en) * | 2014-07-04 | 2014-09-17 | 哈尔滨工业大学深圳研究生院 | Distributed cache method and system |
| CN107071059A (en) * | 2017-05-25 | 2017-08-18 | 腾讯科技（深圳）有限公司 | Distributed caching service implementing method, device, terminal, server and system |
| CN110555041A (en) * | 2018-03-30 | 2019-12-10 | 腾讯科技（深圳）有限公司 | Data processing method, data processing device, computer equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| MIKIO KATAOKA ET AL: "Distributed Cache System for Large-Scale Networks" * |
| 鲍远松 等: "一种分布式时序数据库的存储架构设计及实现方法" * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112487028A (en) * | 2020-11-09 | 2021-03-12 | 杭州电魂网络科技股份有限公司 | Method, system, electronic device and storage medium for server data storage |
| CN112333036A (en) * | 2021-01-06 | 2021-02-05 | 广州技象科技有限公司 | Multi-storage-node-based power Internet of things configuration data backup method and device |
| CN112333036B (en) * | 2021-01-06 | 2021-03-30 | 广州技象科技有限公司 | Multi-storage-node-based power Internet of things configuration data backup method and device |
| CN115269660A (en) * | 2022-09-26 | 2022-11-01 | 平安银行股份有限公司 | Cache data processing method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111400350B (en) | 2023-05-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112118565B (en) | Multi-tenant service gray level publishing method, device, computer equipment and storage medium | |
| EP3428811B1 (en) | Database interface agent for a tenant-based upgrade system | |
| JP6553822B2 (en) | Dividing and moving ranges in distributed systems | |
| JP7221975B2 (en) | Transaction processing in a multi-master distributed data management system | |
| US10956374B2 (en) | Data recovery method, apparatus, and system | |
| US10936423B2 (en) | Enhanced application write performance | |
| CN111400350B (en) | Configuration data reading method, system, electronic device and storage medium | |
| CN102938784A (en) | Method and system used for data storage and used in distributed storage system | |
| JP2017509990A (en) | Reduced data volume durability status for block-based storage | |
| CN111897638A (en) | Distributed task scheduling method and system | |
| CN110753084A (en) | Uplink data reading method, cache server and computer readable storage medium | |
| JP2021149409A (en) | Application development support system and application development support method | |
| CN113961510A (en) | File processing method, device, equipment and storage medium | |
| CN111338834A (en) | Data storage method and device | |
| CN111414356A (en) | Data storage method and device, non-relational database system and storage medium | |
| CN113791798B (en) | Model updating method and device, computer storage medium and electronic equipment | |
| US11044298B1 (en) | Data provider agnostic change handling in mobile client applications | |
| CN111125257B (en) | Dictionary updating method, device, equipment and storage medium | |
| US20200387412A1 (en) | Method To Manage Database | |
| US11693844B2 (en) | Processing delete requests based on change feed of updates | |
| CN113360689B (en) | Image retrieval system, method, related device and computer program product | |
| CN115510016A (en) | Client response method, device and medium based on directory fragmentation | |
| CN115238006A (en) | Retrieval data synchronization method, device, equipment and computer storage medium | |
| CN114547108A (en) | Data processing method, device, equipment and medium | |
| US11416468B2 (en) | Active-active system index management |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |