CN114519082A - Data processing method and related device - Google Patents
Data processing method and related device Download PDFInfo
- Publication number
- CN114519082A CN114519082A CN202210157741.5A CN202210157741A CN114519082A CN 114519082 A CN114519082 A CN 114519082A CN 202210157741 A CN202210157741 A CN 202210157741A CN 114519082 A CN114519082 A CN 114519082A
- Authority
- CN
- China
- Prior art keywords
- data
- cache
- service
- data source
- interface
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/955—Retrieval from the web using information identifiers, e.g. uniform resource locators [URL]
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The application discloses a data processing method and a related device, wherein a service request which is sent by a terminal and comprises an interface name and a target parameter is obtained, the target parameter comprises the interface parameter, and configuration information is obtained by searching based on the interface name and the interface parameter; and assembling to obtain the context text content according to the configuration information. Determining a URL address of a data source to be requested according to the target parameters and the configuration information; generating a cache identifier according to the URL address of the data source to be requested; inquiring cache data in the universal cache system according to the cache identification; if the cache data exist and the service expiration of the cache data is determined according to the data source information, returning the cache data to the terminal, and requesting a corresponding data source interface to acquire the service data through an asynchronous thread according to the URL address of the data source to be requested so as to update the acquired service data to the cache corresponding to the cache identifier. The scheme improves the processing efficiency of the service request, guarantees the real-time performance of the service data and improves the user experience.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data processing method and a related apparatus.
Background
When a user uses internet service, the response speed of a server greatly influences the user experience, fast response data is needed to provide high-availability and high-performance internet experience for the user, and in order to respond to the data fast, a common practice in the industry is to Cache business data, for example, using a caching middleware (Redis, Memcached and the like), and may also combine with mechanisms such as Cache _ Control, ETag, Expires, Last-Modified and the like of a hypertext Transfer Protocol (HTTP), but since the business data can be updated in real time according to the business situation, the above-mentioned technology cannot guarantee the real-time performance of the business data.
Therefore, how to improve the processing efficiency of the terminal service request and ensure the real-time performance of the data is a technical problem that needs to be solved urgently.
Disclosure of Invention
In order to solve the above technical problem, the present application provides a data processing method and a related apparatus, where in a case of service data update, cache data may be returned to a terminal first, and then a latest service data update cache is obtained from a data source through an asynchronous thread, and the update of the service data does not affect the return of the cache data to the terminal, so as to improve the processing efficiency of a service request, ensure the real-time performance of the service data, and improve user experience.
In a first aspect, an embodiment of the present application provides a data processing method, where the method includes:
acquiring a service request sent by a terminal, wherein the service request comprises an interface name and target parameters of a service interface of a universal cache system, and the target parameters comprise interface parameters;
searching and obtaining configuration information based on the interface name and the interface parameter;
assembling and obtaining context text content according to the configuration information, wherein the context text content comprises the configuration information, the service request and data source information associated with the service request;
determining a URL address of a data source to be requested according to the target parameters and the configuration information in the service request;
generating a cache identifier according to the URL address of the data source to be requested;
inquiring cache data according to the cache identification;
if the cache data exists and the service expiration of the cache data is determined according to the data source information, returning the cache data to the terminal, and requesting a corresponding data source interface to acquire service data through an asynchronous thread according to the URL address of the data source to be requested so as to update the acquired service data to the cache corresponding to the cache identifier.
Optionally, the determining, according to the target parameter in the service request and the configuration information, the URL address of the data source to be requested includes:
if the third-party basic service is judged to be depended on according to the information whether the third-party basic service is depended on, adding a response result depended on by the third-party basic service into the target parameter of the service request, wherein the target parameter also comprises the response result correspondingly;
replacing the dynamic parameters in the interface address of the data source to be requested with the values of the target parameters and the values corresponding to the target configuration items in the configuration information respectively to obtain the URL address of the data source to be requested;
the generating of the cache identifier according to the URL address of the data source to be requested includes:
and replacing the URL address to generate the cache identifier.
Optionally, the determining that the service of the cache data is expired according to the data source information includes:
checking whether the system time corresponding to the cache data is greater than or equal to the sum of the update time of the cache data and the service validity period in the data source information;
If yes, determining that the cache data generation service is overdue.
Optionally, the asynchronous thread has a lock released at regular time, and the lock is released after the acquired service data is updated to the cache corresponding to the cache identifier.
Optionally, the method further includes:
if the cache data does not exist, accessing a corresponding data source interface according to the URL address of the data source to be requested to acquire service data;
and returning the service data to the terminal, and storing the service data into a cache corresponding to the cache identifier.
Optionally, the method further includes:
and if the cached data exists and the cached data is determined not to be out of date by service, returning the cached data to the terminal.
In a second aspect, an embodiment of the present application provides a data processing apparatus, where the apparatus includes an obtaining unit, a searching unit, an assembling unit, a determining unit, a generating unit, an inquiring unit, a returning unit, and an updating unit:
the acquiring unit is used for acquiring a service request sent by a terminal, wherein the service request comprises an interface name and target parameters of a service interface of a general cache system, and the target parameters comprise interface parameters;
The searching unit is used for searching and obtaining configuration information based on the interface name and the interface parameter;
the assembling unit is configured to assemble context text content according to the configuration information, where the context text content includes the configuration information, the service request, and data source information associated with the service request;
the determining unit is used for determining the URL address of the data source to be requested according to the target parameter and the configuration information in the service request;
the generating unit is used for generating a cache identifier according to the URL address of the data source to be requested;
the query unit is used for querying the cache data according to the cache identification;
the returning unit is used for returning the cache data to the terminal if the cache data exists and the cache data generation service is determined to be overdue according to the data source information;
and the updating unit is used for requesting a corresponding data source interface to acquire service data through an asynchronous thread according to the URL address of the data source to be requested so as to update the acquired service data to the cache corresponding to the cache identifier.
Optionally, the configuration information includes information about whether to depend on a third-party basic service and an interface address of the data source to be requested, and the determining unit is configured to:
if the third-party basic service is judged to be depended on according to the information whether the third-party basic service is depended on, adding a response result depended on by the third-party basic service into the target parameter of the service request, wherein the target parameter also comprises the response result correspondingly;
replacing dynamic parameters in the interface address of the data source to be requested with values corresponding to the target parameters and the target configuration items in the configuration information respectively to obtain a URL address of the data source to be requested;
the generating unit is configured to:
and replacing the URL address to generate the cache identifier.
Optionally, the query unit is specifically configured to:
checking whether the system time corresponding to the cache data is greater than or equal to the sum of the update time of the cache data and the service validity period in the data source information;
if yes, determining that the cache data generation service is overdue.
Optionally, the asynchronous thread has a lock released at regular time, and the lock is released after the acquired service data is updated to the cache corresponding to the cache identifier.
Optionally, the apparatus further includes an obtaining unit and a storage unit:
the acquisition unit is used for accessing a corresponding data source interface to acquire service data according to the URL address of the data source to be requested if the cache data does not exist as inquired by the inquiry unit;
the return unit is used for returning the service data to the terminal;
the storage unit is configured to store the service data in a cache corresponding to the cache identifier.
Optionally, the returning unit is further configured to:
and if the query unit queries that the cache data exists and determines that the cache data is not out of service, returning the cache data to the terminal.
In a third aspect, an embodiment of the present application provides an electronic device for data processing, where the electronic device includes a processor and a memory:
the memory is used for storing program codes and transmitting the program codes to the processor;
the processor is configured to perform the method of any of the first aspect according to instructions in the program code.
In a fourth aspect, embodiments of the present application provide a computer-readable storage medium for storing program code, which when executed by a processor causes the processor to perform the method of any one of the first aspect.
According to the technical scheme, the embodiment of the application has the following advantages:
acquiring a service request sent by a terminal, wherein the service request comprises an interface name and a target parameter of a service interface of a general cache system, and the target parameter comprises an interface parameter; searching and obtaining configuration information based on the interface name and the interface parameter; assembling to obtain context text content according to the configuration information, wherein the context text content comprises configuration information, a service request and data source information associated with the service request; determining the URL address of a data source to be requested according to the target parameters and the configuration information in the service request; generating a cache identifier according to the URL address of the data source to be requested; inquiring cache data in the universal cache system according to the cache identification; if the cache data exist and the service expiration of the cache data is determined according to the data source information, returning the cache data to the terminal, and requesting a corresponding data source interface to acquire the service data through an asynchronous thread according to the URL address of the data source to be requested so as to update the acquired service data to the cache corresponding to the cache identifier. According to the method, under the condition of service data updating, the cache data can be returned to the terminal, then the latest service data updating cache is obtained from the data source through the asynchronous thread, and the updating of the service data does not affect the return of the cache data to the terminal, so that the processing efficiency of service requests is improved, the real-time performance of the service data is guaranteed, and the user experience is improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a hardware scene diagram of a data processing method according to an embodiment of the present application;
fig. 2 is a flowchart of a data processing method according to an embodiment of the present application;
fig. 3 is a flowchart of another data processing method provided in the embodiment of the present application;
fig. 4 is a structural diagram of a data processing apparatus according to an embodiment of the present application.
Detailed Description
In order to make the technical solutions of the present application better understood, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Because the service data can be updated in real time according to the service condition, the processing efficiency of the service request of the current caching mechanism terminal is low, and the real-time performance of the service data cannot be guaranteed.
Therefore, the embodiment of the present application provides a data processing method, which may obtain a service request sent by a terminal, where the service request includes an interface name and a target parameter of a service interface of a universal cache system, and the target parameter includes an interface parameter; searching and obtaining configuration information based on the interface name and the interface parameter; assembling to obtain context text content according to the configuration information, wherein the context text content comprises the configuration information, the service request and data source information associated with the service request; determining the URL address of a data source to be requested according to the target parameters and the configuration information in the service request; generating a cache identifier according to the URL address of the data source to be requested; inquiring cache data in the universal cache system according to the cache identification; if the cache data exist and the service expiration of the cache data is determined according to the data source information, returning the cache data to the terminal, and requesting a corresponding data source interface to acquire the service data through an asynchronous thread according to the URL address of the data source to be requested so as to update the acquired service data to the cache corresponding to the cache identifier. According to the method, under the condition of service data updating, the cache data can be returned to the terminal, then the latest service data updating cache is obtained from the data source through the asynchronous thread, and the updating of the service data does not affect the return of the cache data to the terminal, so that the processing efficiency of service requests is improved, the real-time performance of the service data is guaranteed, and the user experience is improved.
It should be noted that the method provided in the embodiment of the present application may be applied to a scenario in which a universal cache system interacts with a terminal, where the terminal may be a smart phone, a Personal Digital Assistant (PDA), a tablet computer, a Mobile Internet Device (MID), a Wearable Device (Wearable Device), or the like. The universal caching system may be integrated on the electronic device.
Next, a data processing method provided in the embodiment of the present application will be described with reference to an application scenario.
Referring to fig. 1, fig. 1 shows an architecture diagram of an application scenario, which includes a terminal 101, a general cache system 102, and a data source 103, where the general cache system 102 may be integrated on an electronic device, and the electronic device may be a terminal, a server, or the like. The data source 103 may include a plurality of data sources, such as data source one, data source two, … …, data source N in FIG. 1.
The terminal 101 may send a service request to the universal cache system 102, where the service request includes an interface name and target parameters of a service interface of the universal cache system, and the target parameters include interface parameters. After obtaining the service request, the generic cache system 102 may search for configuration information based on the interface name and the interface parameter.
The universal cache system 102 assembles context text content according to the configuration information, where the context text content includes the configuration information, the service request, and data source information associated with the service request. The universal cache system 102 determines the URL address of the data source to be requested according to the target parameters and configuration information in the service request. The data source to be requested may be any data source in the data sources 103, for example, the data source to be requested may be a data source one. The data source information may be information related to a to-be-requested data source that needs to be requested by the service request, and may include, for example, a service validity period, a protocol, and the like, and when a format of service data required by the terminal is different from a format of service data in the data source, format conversion may be performed based on the protocol.
The universal cache system 102 generates a cache identifier according to the URL address of the data source to be requested, and queries cache data according to the cache identifier. If the existence of the cache data is inquired and the service expiration of the cache data is determined according to the data source information, returning the cache data to the terminal, and requesting a corresponding data source interface to acquire service data through an asynchronous thread according to the URL address of the data source to be requested (for example, if the data source to be requested is a data source I, the data source interface of the data source I is requested), so as to update the acquired service data to the cache corresponding to the cache identifier.
Next, a data processing method provided by an embodiment of the present application will be described in detail with reference to the accompanying drawings. Referring to fig. 2, the method includes:
s201, acquiring a service request sent by a terminal, wherein the service request comprises an interface name and target parameters of a service interface of a general cache system, and the target parameters comprise interface parameters.
S202, searching and obtaining configuration information based on the interface name and the interface parameter.
The universal cache system receives a service request sent by a terminal through a service interface of the universal cache system, wherein the service request comprises an interface name and an interface parameter of the service interface of the universal cache system. The specific configuration information of the current service request is obtained according to the interface name, the interface parameters and the parameter values of the service configuration, and the configuration information may include an interface address (for example, REST interface address) of a data source to be requested, a default parameter name and value, information on whether a third-party basic service is relied on, actual expiration time of a cache, and the like.
S203, context text content is obtained through assembling according to the configuration information, and the context text content comprises the configuration information, the service request and data source information related to the service request.
The universal cache system initializes the Context based on the configuration information of the current service request, and generates Context text content (Context content) associated with the service request, wherein the Context text content may record the configuration information, the service request and data source information associated with the service request.
S204, determining the URL address of the data source to be requested according to the target parameter and the configuration information in the service request.
In a possible implementation manner, if the configuration information includes information whether to depend on the third-party basic service and an interface address of the data source to be requested, the implementation manner of S204 may be to determine whether to depend on the third-party basic service according to the information whether to depend on the third-party basic service. Referring to fig. 1, the third-party basic service may include a first third-party basic service, a second third-party basic service, and the like, and determine whether to rely on a certain third-party basic service. And if the dependence on the third-party basic service is judged according to the information of whether the third-party basic service is depended, adding a response result depended by the third-party basic service into the target parameter of the service request. The response result that the third party basic service depends on can be obtained, for example, using a parallel asynchronous execution of RXJava. At this time, the target parameter includes an interface parameter and a response result, and the target parameter may be represented by a parameter Map. And then, replacing the dynamic parameters in the interface address of the data source to be requested with values corresponding to the target parameters and the target configuration items in the configuration information respectively to obtain the URL address of the data source to be requested.
And S205, generating a cache identifier according to the URL address of the data source to be requested.
In this embodiment of the present application, the manner of generating the cache identifier according to the URL address of the data source to be requested may be to replace the URL address to generate the cache identifier. For example, if "/" is included in the URL address, then all of the "/" may be replaced with "_", thereby obtaining a cache id. Wherein the cache identifier may be a cache key (key).
And S206, inquiring the cache data according to the cache identification.
S207, if the cache data is found and the cache data is determined to be out of date according to the data source information, returning the cache data to the terminal, and requesting a corresponding data source interface to acquire service data through an asynchronous thread according to the URL address of the data source to be requested so as to update the acquired service data to a cache corresponding to the cache identifier.
The general cache system uses a cache key to inquire cache middleware, the cache middleware is divided into local caches to store hot service data, a centralized cache stores all service data, and if the cache data exists, whether the cache data is out of date or not is determined according to data source information. And if the service of the cache data is determined to be overdue, returning the cache data to the terminal, and requesting a corresponding data source interface to acquire the service data through an asynchronous thread according to the URL address of the data source to be requested so as to update the acquired service data to a cache corresponding to the cache identifier.
In a possible implementation manner, the manner of determining that the service of the cache data is out of date according to the data source information may be to check whether system time (systemTime) corresponding to the cache data is greater than or equal to a sum (expireTime) of update time (updateTime) of the cache data and a service validity period in the data source information, and if yes, determine that the service of the cache data is out of date, thereby initiating an asynchronous thread to call a URL address of a data source to be requested to update the cache data. If the cached data exists and the cached data is determined to be out of date of service, the cached data is directly returned to the terminal.
It should be noted that, the asynchronous thread has a lock released at regular time, the thread taking the lock may continue to execute the task of calling the URL address of the data source to be requested to update the cache data, and the thread not taking the lock is released immediately. And releasing the lock after the acquired service data is updated to the cache corresponding to the cache identifier. If abnormal, the lock is released after a specified time is exceeded.
The mechanism ensures that only one thread of hot service data updates the cache data of the data source in a single process, thereby reducing the pressure of data source service.
And the asynchronous thread which takes the executable lock can acquire the Context of the service thread transmitted in, directly initiate a request according to data source information in the Context, and immediately retry once by using a SpringRetry component if the request fails. When the data source interface returns normally, completely updating the cache data; when the data source interface returns to be abnormal, if the status code of the data source interface is 5xx (service unavailable) or overtime, only updating expireTime in the cache data for 10 seconds to promote that the next service request can immediately update the cache data again; if the data source interface returns to 404, which means that the cache data is deleted, businessData of the cache data is reset to null, and fields such as time udateTime for acquiring service data are updated as usual.
In some possible cases, if the cache data does not exist, the corresponding data source interface is accessed according to the URL address of the data source to be requested to obtain the service data, the service data is returned to the terminal, and the service data is stored in the cache corresponding to the cache identifier, for example, in the cache middleware. The service data returned by the data source interface should include creation time of the service data, validity time of the service data, and concrete service data businessData output to the user, and at the same time, the system time when the service data is currently acquired needs to be recorded as update time updateTime.
According to the technical scheme, the embodiment of the application has the following advantages:
acquiring a service request sent by a terminal, wherein the service request comprises an interface name and a target parameter of a service interface of a universal cache system, and the target parameter comprises an interface parameter; searching and obtaining configuration information based on the interface name and the interface parameter; assembling to obtain context text content according to the configuration information, wherein the context text content comprises the configuration information, the service request and data source information associated with the service request; determining the URL address of a data source to be requested according to the target parameters and the configuration information in the service request; generating a cache identifier according to the URL address of the data source to be requested; inquiring cache data in the universal cache system according to the cache identification; if the cache data exist and the service expiration of the cache data is determined according to the data source information, returning the cache data to the terminal, and requesting a corresponding data source interface to acquire the service data through an asynchronous thread according to the URL address of the data source to be requested so as to update the acquired service data to the cache corresponding to the cache identifier. According to the method, under the condition of service data updating, the cache data can be returned to the terminal, then the latest service data updating cache is obtained from the data source through the asynchronous thread, and the updating of the service data does not affect the return of the cache data to the terminal, so that the processing efficiency of service requests is improved, the real-time performance of the service data is guaranteed, and the user experience is improved.
In addition, the parameter in the REST PATH is agreed with the data source to serve as the key parameter for influencing the result, then the PATH specific value in the URL address can be directly used as the cache key, the service validity period in the cache data is specified by the data source, and the asynchronous thread immediately initiates the data source interface request and updates the service data in the cache after the cache data is determined to be out of service.
The general cache system faces to the terminal, most service requests are directly returned by cache data, and the response speed is accelerated. The data source only needs to bear the service request of data updating, and the service request quantity is small. The data source access with weak concurrency capability can provide more stable and faster user experience for the user. When the universal cache system accesses a new data source, coding is not needed, and only rule configuration is completed according to business requirements, so that the access flow of the new data source is simplified.
An embodiment of the present application further provides another data processing method, and as shown in fig. 3, fig. 3 shows a flowchart of another data processing method, where the method includes:
s301, receive service request/data/lista equals 1& b equals 2.
S302, configuration information is searched according to the service request.
The general cache system receives a service request sent by a terminal: and then searching configuration information according to the interface name of the data/list in the service request, wherein the interface parameters a are 1, and the interface parameters b are 2.
Examples of configuration information are as follows:
the method includes the steps that a requestKey in configuration information is a path of an interface terminal, a requestSubKey is sharing configuration of parameters and values of the terminal (the same interface path can be divided into a plurality of different data source paths according to the parameters), and specific configuration information is located, wherein a targetUri is an interface address of a data source to be requested, key parameters agreed by using REST style exist in the path, targetTimeout is an overtime which is agreed to be used when the data source is directly requested, wherein needCheckParamputhEndaffeltValue is an array which is appointed to check the parameters of the terminal, the values do not accord with default values behind the terminals when expected, the default values behind the parameters are defined by SPEL expressions, and various conditional statements can be dynamically executed. The needUseThidpartPartypParam is information for defining whether the third-party basic service is depended on, the third-party basic service realizes calling logic in the universal cache system, response results returned by the third-party basic service are merged into service requests of the terminal when the third-party basic service depends on, the response results and the interface parameters form target parameters, the remaining configuration item expireTimeWithFail is service cache expiration time when a request data source is abnormal, and expireTimeWithDefault is physical cache expiration time when the request data source is normal.
And S303, assembling to obtain the context text content according to the configuration information.
S304, judging whether the third-party basic service is depended on, if so, executing S305, and if not, executing S306.
S305, obtaining a response result of third-party basic service dependence.
S306, checking the value of the needCheckParamWithDefaultValue based on the generated context text content and the configuration information.
Based on the generated context text content, in combination with internal configuration information, checking a value of a needchecksaramwithdefaultvalue, checking a parameter for each of configuration items included in the configuration information, if the parameter has no valid value, assigning the parameter with a default value in the configuration information, and if the parameter has no valid value, performing execution to obtain a specific value, such as argv, with a Spring engine if the value is configured with a SPEL expression, { # straId? 'space', the value of straId is judged first, and when the value is empty, the value is assigned as space.
S307, assembling the URL address of the data source to be requested according to all the parameters.
And replacing dynamic variables in the interface address of the REST-style data source to be requested defined in the data source with actual parameter values (for example, values corresponding to the target parameters and the target configuration items in the configuration information respectively) by using the UriComponentsBuilder of Spring according to all the parameters (for example, the target parameters including the interface parameters and the response results and other parameters in the configuration information, such as the target configuration items), so as to obtain the complete URL address of the data source to be requested.
For example, the interface address of the REST-style data source to be requested defined in the data source is http:// datasource. com/api/v1/{ a }/{ b }/{ ab }/{ argv }, wherein the dynamic variables include a, b, ab, argv, and the actual parameter values include the values of the interface parameters in the service request: and a is 1 and b is 2, ab is 0 in the configuration information, and argv is space in the configuration information. Therefore, replacing the dynamic variable with the actual parameter value results in the URL address of the data source to be requested being http:// datasource. com/api/v 1/1/2/0/space.
S308, replacing all "/" in the path of the URL address with "_", and obtaining a cache identifier.
All "/" in the path in http:// datasource. com/api/v1/1/2/0/space are replaced by "_", resulting in the cache identification as _ api _ v1_1_2_0_ space.
And S309, inquiring the cache data according to the cache identification.
S310, determining whether the cache data exists, if so, executing S311, and if not, executing S312.
S311, determining whether the service of the cache data is overdue, if yes, executing S313, and if not, executing S314.
And S312, directly requesting the data source interface to acquire the service data and updating the service data into the cache.
And S313, returning the cache data to the terminal, and updating the cache data through the asynchronous thread.
And S314, returning the cached data to the terminal.
If the cache data exists, checking the system time of the cache data as updateTime + expireTime, if the cache data is true, the cache data is overdue, returning the cache data to the terminal immediately, and initiating an asynchronous thread to update the cache data, wherein the asynchronous thread ensures that only one thread executes an update task at the same time by a process level lock; and if the cache data does not exist, immediately requesting the data source interface to acquire the service data and updating the service data into the cache.
Next, a detailed description will be given of an update method of cache data.
Step 1, after the cache identification is obtained, whether cache data exist in the cache middleware is checked according to the cache identification, if the cache data exist, the step 2 is carried out, and if the cache data do not exist, the step 3 is carried out.
Step 2, immediately returning the businessData part in the cache data to the terminal, initiating an asynchronous thread to check whether the cache data is out of date with the service, and not needing any operation if the cache data is not out of date with the service; if the service is overdue, trying to acquire a process-level lock, continuing subsequent execution of the acquired thread, stopping execution of the thread which does not acquire the lock, updating a data source, handing the thread which takes the lock to execute, initiating a request data source interface by the thread, and performing three kinds of processing according to response of the data source interface: when the data source interface responds to 200, caching the service data returned by the data source interface, setting the service validity period of the service data according to expireTime set by the data source, and storing the data returned by the data source interface for 3 days by default (namely, the caching validity period is 3 days); when the data source interface responds 404, it means that the data source has deleted the service data, clears businessData in the cache data, sets the service validity period of the service data according to the expireTime set by the data source, and stores the data returned by the data source interface in a 3-day cache by default (i.e. the cache validity period is 3 days); if the data source interface returns 5xx status codes such as 502, 503 and 504 or no response, the cache data is not changed, the service validity period expireTime of the service data is changed to 10 seconds, and the cache data is stored and updated to 3 days (namely the cache validity period is 3 days).
Step 3, if no corresponding cache data exists in the cache middleware, immediately initiating a data source interface request, and performing three kinds of processing according to interface response: when the data source interface responds to the request 200, caching data returned by the data source interface, setting the service validity period of the service data according to expireTime set by the data source, and storing the data returned by the data source interface for 3 days by default (namely, the caching validity period is 3 days); if the data source interface responds 404, it means that the data source has deleted the service data, clears businessData in the cache data, sets the service validity period of the service data according to the expireTime set by the data source, and stores the data returned by the data source interface for 3 days as default (i.e. the validity period of the cache is 3 days); if the data source interface returns 5xx status codes such as 502, 503 and 504 or no response exists, the cache data is set to be null, the service validity period expireTime of the service data is set to be 10 seconds, and the cache data is stored and updated for 3 days (namely the cache validity period is 3 days).
Based on the data processing method provided by the foregoing embodiment, an embodiment of the present application provides a data processing apparatus, referring to fig. 4, the apparatus includes an obtaining unit 401, a searching unit 402, an assembling unit 403, a determining unit 404, a generating unit 405, an inquiring unit 406, a returning unit 407, and an updating unit 408:
The acquiring unit 401 is configured to acquire a service request sent by a terminal, where the service request includes an interface name and a target parameter of a service interface of a universal cache system, and the target parameter includes an interface parameter;
the searching unit 402 is configured to search for configuration information based on the interface name and the interface parameter;
the assembling unit 403 is configured to assemble, according to the configuration information, context text content, where the context text content includes the configuration information, the service request, and data source information associated with the service request;
the determining unit 404 is configured to determine, according to the target parameter and the configuration information in the service request, a URL address of a data source to be requested;
the generating unit 405 is configured to generate a cache identifier according to the URL address of the data source to be requested;
the query unit 406 is configured to query the cache data according to the cache identifier;
the returning unit 407 is configured to return the cached data to the terminal if it is found that the cached data exists and it is determined that the service of the cached data is expired according to the data source information;
the updating unit 408 is configured to acquire service data through a data source interface corresponding to the asynchronous thread request according to the URL address of the data source to be requested, so as to update the acquired service data to the cache corresponding to the cache identifier.
Optionally, the configuration information includes information on whether to depend on a third-party basic service and an interface address of the data source to be requested, and the determining unit is configured to:
if the dependence on the third-party basic service is judged according to the information whether the dependence on the third-party basic service is judged, adding a response result that the third-party basic service depends on into the target parameter of the business request, wherein the target parameter also comprises the response result correspondingly;
replacing the dynamic parameters in the interface address of the data source to be requested with the values of the target parameters and the values corresponding to the target configuration items in the configuration information respectively to obtain the URL address of the data source to be requested;
the generating unit is configured to:
and replacing the URL address to generate the cache identifier.
Optionally, the query unit is specifically configured to:
checking whether the system time corresponding to the cache data is greater than or equal to the sum of the update time of the cache data and the service validity period in the data source information;
if yes, determining that the cache data generation service is overdue.
Optionally, the asynchronous thread has a lock released at regular time, and the lock is released after the acquired service data is updated to the cache corresponding to the cache identifier.
Optionally, the apparatus further includes an obtaining unit and a storage unit:
the acquisition unit is used for accessing a corresponding data source interface to acquire service data according to the URL address of the data source to be requested if the cache data does not exist as inquired by the inquiry unit;
the return unit is used for returning the service data to the terminal;
the storage unit is configured to store the service data in a cache corresponding to the cache identifier.
Optionally, the returning unit is further configured to:
and if the query unit queries that the cache data exists and determines that the cache data is not out of date in service, returning the cache data to the terminal.
According to the technical scheme, the embodiment of the application has the following advantages:
acquiring a service request sent by a terminal, wherein the service request comprises an interface name and a target parameter of a service interface of a general cache system, and the target parameter comprises an interface parameter; searching and obtaining configuration information based on the interface name and the interface parameter; assembling to obtain context text content according to the configuration information, wherein the context text content comprises configuration information, a service request and data source information associated with the service request; determining the URL address of a data source to be requested according to the target parameters and the configuration information in the service request; generating a cache identifier according to the URL address of the data source to be requested; inquiring cache data in the universal cache system according to the cache identification; if the cache data exist and the service expiration of the cache data is determined according to the data source information, returning the cache data to the terminal, and requesting a corresponding data source interface to acquire the service data through an asynchronous thread according to the URL address of the data source to be requested so as to update the acquired service data to the cache corresponding to the cache identifier. According to the method, under the condition of service data updating, the cache data can be returned to the terminal, then the latest service data updating cache is obtained from the data source through the asynchronous thread, and the updating of the service data does not affect the return of the cache data to the terminal, so that the processing efficiency of service requests is improved, the real-time performance of the service data is guaranteed, and the user experience is improved.
An embodiment of the present application provides an electronic device for data processing, where the electronic device includes a processor and a memory:
the memory is used for storing program codes and transmitting the program codes to the processor;
the processor is configured to perform the method according to any one of the corresponding embodiments of fig. 2 according to instructions in the program code.
The embodiment of the present application provides a computer-readable storage medium for storing program code, and the program code, when executed by a processor, causes the processor to execute the method according to any one of the corresponding embodiments in fig. 2.
Those of ordinary skill in the art will understand that: all or part of the steps for realizing the method embodiments can be completed by hardware related to program instructions, the program can be stored in a computer readable storage medium, and the program executes the steps comprising the method embodiments when executed; and the aforementioned storage medium may be at least one of the following media: various media that can store program codes, such as read-only memory (ROM), RAM, magnetic disk, or optical disk.
It should be noted that, in the present specification, all the embodiments are described in a progressive manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the apparatus and system embodiments, since they are substantially similar to the method embodiments, they are described in a relatively simple manner, and reference may be made to some of the descriptions of the method embodiments for related points. The above-described embodiments of the apparatus and system are merely illustrative, and the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
The above description is only one specific embodiment of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present application should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A method of data processing, the method comprising:
acquiring a service request sent by a terminal, wherein the service request comprises an interface name and target parameters of a service interface of a general cache system, and the target parameters comprise interface parameters;
searching and obtaining configuration information based on the interface name and the interface parameter;
assembling context text content according to the configuration information, wherein the context text content comprises the configuration information, the service request and data source information associated with the service request;
determining a URL address of a data source to be requested according to the target parameter and the configuration information in the service request;
generating a cache identifier according to the URL address of the data source to be requested;
inquiring cache data according to the cache identification;
If the cache data exist and the cache data is determined to be out of date according to the data source information, returning the cache data to the terminal, and requesting a corresponding data source interface to acquire service data through an asynchronous thread according to the URL address of the data source to be requested so as to update the acquired service data to a cache corresponding to the cache identifier.
2. The method according to claim 1, wherein the configuration information includes information about whether to depend on a third-party basic service and an interface address of the data source to be requested, and the determining a URL address of the data source to be requested according to the target parameter in the service request and the configuration information includes:
if the third-party basic service is judged to be depended on according to the information whether the third-party basic service is depended on, adding a response result depended on by the third-party basic service into the target parameter of the service request, wherein the target parameter also comprises the response result correspondingly;
replacing dynamic parameters in the interface address of the data source to be requested with values corresponding to the target parameters and the target configuration items in the configuration information respectively to obtain a URL address of the data source to be requested;
The generating of the cache identifier according to the URL address of the data source to be requested includes:
and replacing the URL address to generate the cache identifier.
3. The method of claim 1, wherein the determining that the buffered data has expired according to the data source information comprises:
checking whether the system time corresponding to the cache data is greater than or equal to the sum of the update time of the cache data and the service validity period in the data source information;
if yes, determining that the cache data generation service is overdue.
4. The method according to any of claims 1-3, wherein the asynchronous thread has a lock released at a certain time, and the lock is released after the acquired service data is updated to the cache corresponding to the cache identifier.
5. The method of claim 1, further comprising:
if the cache data does not exist, accessing a corresponding data source interface according to the URL address of the data source to be requested to acquire service data;
and returning the service data to the terminal, and storing the service data into a cache corresponding to the cache identifier.
6. The method of claim 1, further comprising:
and if the cached data exists and the cached data is determined not to be out of date, returning the cached data to the terminal.
7. A data processing apparatus is characterized by comprising an acquisition unit, a search unit, an assembly unit, a determination unit, a generation unit, a query unit, a return unit and an update unit:
the acquiring unit is used for acquiring a service request sent by a terminal, wherein the service request comprises an interface name and target parameters of a service interface of a universal cache system, and the target parameters comprise interface parameters;
the searching unit is used for searching and obtaining configuration information based on the interface name and the interface parameter;
the assembling unit is configured to assemble context text content according to the configuration information, where the context text content includes the configuration information, the service request, and data source information associated with the service request;
the determining unit is used for determining the URL address of the data source to be requested according to the target parameter and the configuration information in the service request;
The generating unit is used for generating a cache identifier according to the URL address of the data source to be requested;
the query unit is used for querying the cache data according to the cache identification;
the return unit is used for returning the cache data to the terminal if the cache data is found by the query unit and the cache data generation service is determined to be overdue according to the data source information;
and the updating unit is used for requesting a corresponding data source interface to acquire service data through an asynchronous thread according to the URL address of the data source to be requested so as to update the acquired service data to the cache corresponding to the cache identifier.
8. The apparatus of claim 7, wherein the configuration information includes information about whether to depend on a third-party basic service and an interface address of the data source to be requested, and the determining unit is configured to:
if the third-party basic service is judged to be depended on according to the information whether the third-party basic service is depended on, adding a response result depended on by the third-party basic service into the target parameter of the service request, wherein the target parameter also comprises the response result correspondingly;
Replacing the dynamic parameters in the interface address of the data source to be requested with the values of the target parameters and the values corresponding to the target configuration items in the configuration information respectively to obtain the URL address of the data source to be requested;
the generating unit is configured to:
and replacing the URL address to generate the cache identifier.
9. An electronic device for data processing, the electronic device comprising a processor and a memory:
the memory is used for storing program codes and transmitting the program codes to the processor;
the processor is configured to perform the method of any of claims 1-6 according to instructions in the program code.
10. A computer-readable storage medium for storing program code, which, when executed by a processor, causes the processor to perform the method of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210157741.5A CN114519082A (en) | 2022-02-21 | 2022-02-21 | Data processing method and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210157741.5A CN114519082A (en) | 2022-02-21 | 2022-02-21 | Data processing method and related device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114519082A true CN114519082A (en) | 2022-05-20 |
Family
ID=81598955
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210157741.5A Pending CN114519082A (en) | 2022-02-21 | 2022-02-21 | Data processing method and related device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114519082A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115292356A (en) * | 2022-07-21 | 2022-11-04 | 中电金信软件有限公司 | Data query method and device and electronic equipment |
-
2022
- 2022-02-21 CN CN202210157741.5A patent/CN114519082A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115292356A (en) * | 2022-07-21 | 2022-11-04 | 中电金信软件有限公司 | Data query method and device and electronic equipment |
| CN115292356B (en) * | 2022-07-21 | 2023-06-16 | 中电金信软件有限公司 | Data query method and device and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102097881B1 (en) | Method and apparatus for processing a short link, and a short link server | |
| CN111464615B (en) | Request processing method, device, server and storage medium | |
| US9448932B2 (en) | System for caching data | |
| US8856365B2 (en) | Computer-implemented method, computer system and computer readable medium | |
| CN110557464A (en) | DNS (Domain name Server) resolution method, authoritative DNS server and DNS resolution system | |
| US8352442B2 (en) | Determination of an updated data source from disparate data sources | |
| CN111782692A (en) | Frequency control method and device | |
| CA2845932C (en) | Method and system for registering software systems in data-sharing sessions | |
| US9928174B1 (en) | Consistent caching | |
| CN114519082A (en) | Data processing method and related device | |
| CN113656098B (en) | Configuration acquisition method and system | |
| US10827035B2 (en) | Data uniqued by canonical URL for rest application | |
| CN110233843B (en) | User request processing method and device | |
| CN111767053A (en) | Front-end page data acquisition method and device | |
| CN110807040B (en) | Method, device, equipment and storage medium for managing data | |
| CN115017538A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN113032820B (en) | File storage method, access method, device, equipment and storage medium | |
| CN114840574A (en) | Data query method, server, node and storage medium | |
| CN109088913B (en) | Method for requesting data and load balancing server | |
| CN111367634A (en) | Information processing method, information processing device and terminal equipment | |
| CN110716941A (en) | HANDLE identifier analysis system and data query method | |
| US20240089339A1 (en) | Caching across multiple cloud environments | |
| CN116048468A (en) | Method, device, electronic equipment and medium for loading cache data and processing data | |
| CN113961601A (en) | Request processing method, device, medium and electronic equipment | |
| CN113590661A (en) | Data caching method, device, server and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |