CN112445988A

CN112445988A - Data loading method and device

Info

Publication number: CN112445988A
Application number: CN201910809111.XA
Authority: CN
Inventors: 王占民
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Priority date: 2019-08-29
Filing date: 2019-08-29
Publication date: 2021-03-05

Abstract

The invention discloses a data loading method and device, and relates to the technical field of computers. One embodiment of the method comprises: after receiving a query request for data, checking whether the data requested to be queried is loaded to a cache; if the data is loaded into the cache, returning the cached data; if the query request is not loaded into the cache, checking whether an equivalent query request of the query request exists; if yes, abandoning the query request; otherwise, adding the service ID included in the query request to the loading cache request queue; and inquiring data corresponding to each service ID in the loading cache request queue according to a preset strategy, and loading the inquired data to the cache. The implementation method can avoid long-time waiting of inquirers, save memory, realize the current limitation of the requests, discard excessive requests, avoid the rapid consumption of the computing resources of the server side, control the speed of acquiring data from the data source inquiry, avoid the instantaneous high concurrent access to the data source and prevent the rapid consumption of the computing resources of the data source.

Description

Data loading method and device

Technical Field

The invention relates to the technical field of computers, in particular to a data loading method and device.

Background

The memory cache is a layer of buffer between the data inquirer and the data source, which can not only reduce the pressure of the data source, but also quickly respond to the data inquirer to deal with high concurrent access. Existing data loading schemes include two types: one is that the server synchronously loads a piece of data to the memory cache when the data inquirer inquires a piece of data for the first time, and then provides the data loaded to the cache to the data inquirer, and the scheme ensures that the data inquirer can read the data from the cache only after the cache loading is finished; the other is that the server uses the background task to load all the data into the memory cache in advance, so that the data inquirer is not required to wait for the completion of cache loading, but the scheme is only suitable for the condition of small data volume.

In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:

in the first existing scheme, a data inquirer waits for a long time in the process of loading data into a cache at a server;

in the second conventional scheme, a large amount of data which is possibly needed is loaded to a memory cache in advance, so that more memory space is wasted;

in addition, the prior art lacks a current limiting mechanism for the request, so that the computing resources of the server are rapidly consumed when a large number of query requests exist, and the server is difficult to continuously provide services; and the data inquiry rate from the data source is not limited, so that when a large number of requests are instantly inquired on the data source, the computing resources of the data source are rapidly consumed, and the data source is difficult to continuously provide services.

Disclosure of Invention

In view of this, embodiments of the present invention provide a data loading method and apparatus, which can avoid that a data inquirer waits for a long time, only needs to load data required by the data inquirer, save memory space, limit current for inquiry requests, discard excessive requests, avoid rapid consumption of server computing resources, ensure that the server continuously provides services, and control a rate of acquiring data from a data source, avoid instantaneous high concurrent access to the data source, prevent rapid consumption of data source computing resources, and ensure that the data source continuously provides services.

To achieve the above object, according to an aspect of an embodiment of the present invention, a data loading method is provided.

A method of data loading, comprising: after receiving a query request for data, checking whether the data requested to be queried is loaded to a cache or not, wherein the query request comprises a service ID of the data requested to be queried; if the data requested to be queried is loaded to the cache, returning the cached data requested to be queried; if the data requested to be queried is not loaded to the cache, checking whether an equivalent query request of the query request already exists, wherein the equivalent query request and the query request comprise the same service ID; if so, discarding the query request; otherwise, adding the service ID included in the query request to a loading cache request queue; and querying data corresponding to each service ID in the load cache request queue according to a preset strategy, and loading the queried data to the cache.

Optionally, the step of checking whether there is an equivalent query request of the query request already includes: and checking whether a key which is equal to the service ID included in the query request exists in a mapping, if so, the equivalent query request exists, otherwise, the equivalent query request does not exist, wherein the mapping includes a key-value pair, the key of the key-value pair is the service ID, and the value is a preset mark value.

Optionally, the method further comprises: and under the condition that a key equal to the service ID included in the query request does not exist in the mapping, generating a key-value pair by taking the service ID included in the query request as the key and the preset mark value as the value, and adding the generated key-value pair to the mapping.

Optionally, on a condition that it is checked and determined that the length of the mapping is greater than the first threshold, the operation of adding the service ID included in the query request to the load cache request queue is performed, and the operation of generating a key-value pair with the service ID included in the query request as a key and the preset flag value as a value, and adding the generated key-value pair to the mapping is performed.

Optionally, after the queried data is loaded into the cache, the key-value pair corresponding to the service ID of the queried data in the map is cleared.

Optionally, the step of checking whether there is an equivalent query request of the query request already includes: and traversing the loading cache request queue to check whether the loading cache request queue stores the service ID included by the query request, if so, the equivalent query request exists, otherwise, the equivalent query request does not exist.

Optionally, the operation of adding the service ID included in the query request to the load cache request queue is performed on the condition that it is checked and determined that the length of the load cache request queue is greater than the second threshold.

Optionally, the step of querying data corresponding to each service ID in the load cache request queue according to a preset policy includes: according to configured time intervals, service IDs are regularly acquired from the load cache request queue according to a specific sequence, wherein the number of the service IDs which can be acquired each time is determined through configuration; and inquiring data corresponding to each service ID in the acquired service IDs from the data source.

According to another aspect of the embodiments of the present invention, a data loading apparatus is provided.

A data loading apparatus comprising: the data cache checking module is used for checking whether the data requested to be inquired is loaded to a cache after receiving an inquiry request for the data, wherein the inquiry request comprises a service ID of the data requested to be inquired; the cache data returning module is used for returning the cached data requested to be queried if the data requested to be queried is loaded to the cache; a query request processing module, configured to check whether an equivalent query request of the query request already exists if the data requested to be queried is not loaded to the cache, where the equivalent query request and the query request include the same service ID; if so, discarding the query request; otherwise, adding the service ID included in the query request to a loading cache request queue; and the data query and loading module is used for querying data corresponding to each service ID in the loading cache request queue according to a preset strategy and loading the queried data to the cache.

Optionally, the query request processing module includes a first equivalent query request checking sub-module, configured to: and checking whether a key which is equal to the service ID included in the query request exists in a mapping, if so, the equivalent query request exists, otherwise, the equivalent query request does not exist, wherein the mapping includes a key-value pair, the key of the key-value pair is the service ID, and the value is a preset mark value.

Optionally, the query request processing module is further configured to: and under the condition that a key equal to the service ID included in the query request does not exist in the mapping, generating a key-value pair by taking the service ID included in the query request as the key and the preset mark value as the value, and adding the generated key-value pair to the mapping.

Optionally, the query request processing module, under a condition that it is checked and determined that the length of the mapping is greater than a first threshold, executes the operation of adding the service ID included in the query request to a load cache request queue, and generates a key-value pair with the service ID included in the query request as a key and the preset flag value as a value, and adds the generated key-value pair to the mapping operation.

Optionally, the apparatus further comprises a cleaning module, configured to: and after the inquired data is loaded into the cache, cleaning the corresponding key value pair of the service ID of the inquired data in the mapping.

Optionally, the query request processing module includes a second equivalent query request checking sub-module, configured to: and traversing the loading cache request queue to check whether the loading cache request queue stores the service ID included by the query request, if so, the equivalent query request exists, otherwise, the equivalent query request does not exist.

Optionally, the query request processing module executes the operation of adding the service ID included in the query request to the load cache request queue under the condition that it is checked and determined that the length of the load cache request queue is greater than the second threshold.

Optionally, the data query and load module includes a data query submodule configured to: according to configured time intervals, service IDs are regularly acquired from the load cache request queue according to a specific sequence, wherein the number of the service IDs which can be acquired each time is determined through configuration; and inquiring data corresponding to each service ID in the acquired service IDs from the data source.

According to yet another aspect of an embodiment of the present invention, an electronic device is provided.

An electronic device, comprising: one or more processors; a memory for storing one or more programs, which when executed by the one or more processors, cause the one or more processors to implement the data loading method provided by the present invention.

According to yet another aspect of an embodiment of the present invention, a computer-readable medium is provided.

A computer-readable medium, on which a computer program is stored, which, when executed by a processor, implements the data loading method provided by the present invention.

One embodiment of the above invention has the following advantages or benefits: after receiving the query request for the data, checking whether the data requested to be queried is loaded to a cache or not, and if the data requested to be queried is loaded to the cache, returning the cached data requested to be queried; if the data requested to be inquired is not loaded in the cache, checking whether an equivalent inquiry request of the inquiry request exists, if so, abandoning the inquiry request, otherwise, adding the service ID included in the inquiry request into a loading cache request queue, inquiring the data corresponding to each service ID in the loading cache request queue according to a preset strategy, and loading the inquired data into the cache. The method and the device can avoid long-time waiting of the data inquirer, only need to load the data required by the data inquirer, and save the memory space. The length of mapping or the length of a loading cache request queue is controlled by a threshold value, so that the current limitation of the query request is realized, excessive requests are discarded, the computing resources of the server are prevented from being rapidly consumed, and the server is ensured to continuously provide services. According to the configured time interval, service IDs are regularly acquired from the load cache request queue according to a specific sequence, and the quantity of the service IDs which can be acquired each time is determined through configuration, so that the data acquisition rate from the data source query can be controlled, instantaneous high concurrent access to the data source is avoided, the computing resources of the data source are prevented from being rapidly consumed, and the data source is ensured to continuously provide services.

Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.

Drawings

The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:

FIG. 1 is a schematic diagram of the main steps of a data loading method according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a data loading flow according to one embodiment of the invention;

FIG. 3 is a schematic illustration of a principle of data loading according to another embodiment of the present invention;

FIG. 4 is a schematic diagram of a data loading flow according to yet another embodiment of the invention;

FIG. 5 is a schematic diagram of the main modules of a data loading apparatus according to an embodiment of the present invention;

FIG. 6 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;

fig. 7 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.

Detailed Description

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

As will be appreciated by one skilled in the art, embodiments of the present invention may be embodied as a system, apparatus, device, method, or computer program product. Accordingly, the present disclosure may be embodied in the form of: entirely hardware, entirely software (including firmware, resident software, micro-code, etc.), or a combination of hardware and software.

Fig. 1 is a schematic diagram of main steps of a data loading method according to a first embodiment of the present invention.

As shown in fig. 1, the data loading method of the present embodiment mainly includes the following steps S101 to S103.

Step S101: after each query request for data is received, it is checked whether the data requesting the query has been loaded into the cache.

The query request includes a service ID (identification) of the data requesting the query.

The cache refers to a memory cache of the server, is a space opened up in the memory, and has the characteristic of high response query speed, because the cost is high, too much data cannot be stored in the memory cache.

The embodiment of the present invention does not limit the specific type of the cache, and the cache may be, for example, Redis (an open source support network, a log type that may be based on a memory or may be persistent, a Key-Value database), Memcached (a high-performance distributed memory object cache system), and the like.

The cache has an expiration time (the length of the expiration time may be specified in advance), and that the data is loaded into the cache means that the data exists in the cache and is not expired.

Step S102: if the data requested to be queried is loaded to the cache, returning the cached data requested to be queried; if the data requested to be queried is not loaded to the cache, checking whether an equivalent query request of the query request exists, and if so, discarding the query request; otherwise, the service ID included in the query request is added to the load cache request queue.

Wherein an equivalent of the query request is a request that includes the same service ID as the query request. The two requests including the same service ID means that the two requests are substantially the same request, and in the case where one of the requests already exists, there is no need to repeat the processing of the request received later, and discarding is required.

As an embodiment, the step of checking whether there is an equivalent query request of the query request may specifically include: and checking whether a key which is equal to the service ID included by the query request exists in the mapping, if so, then, existing the equivalent query request of the query request, otherwise, not, existing the equivalent query request, wherein the mapping includes key-value pairs, the key of the key-value pair is the service ID, and the value is a preset mark value.

And generating a key-value pair by taking the service ID included in the inquiry request as a key and a preset mark value as a value when the key equal to the service ID included in the inquiry request does not exist in the mapping, and adding the generated key-value pair to the mapping.

Before adding the service ID included in the query request to the load cache request queue and adding the generated key-value pair to the mapping, checking whether the length of the mapping is greater than a first threshold value, if so, executing an operation of adding the service ID included in the query request to the load cache request queue, and generating the key-value pair and adding the generated key-value pair to the mapping by using the service ID included in the query request as a key and using a preset flag value as a value; if less than or equal to the first threshold, the query request is discarded.

As another embodiment, the step of checking whether there is an equivalent query request of the query request may specifically include: and traversing the load cache request queue to check whether the load cache request queue stores the service ID included by the query request, if so, existing the equivalent query request of the query request, and otherwise, not existing the equivalent query request.

Before adding the service ID included in the query request to the load cache request queue, checking whether the length of the load cache request queue is greater than a second threshold, and performing an operation of adding the service ID included in the query request to the load cache request queue under a condition that it is determined that the length of the load cache request queue is greater than the second threshold; if the length of the load cache request queue is less than or equal to the second threshold, the query request is discarded.

The service ID included in the query request is added to the load cache request queue, specifically, the service ID included in the query request is added to the tail (or queue tail) of the load cache request queue. A queue is a special linear table that allows only delete operations at the head of the table (i.e., the head of the queue) and insert operations at the tail of the table (i.e., the tail of the queue).

The above two embodiments will be described in detail below, with particular reference to the corresponding examples of fig. 2 to 4.

After discarding the query request, a prompt message may be output, such as a prompt to the data querier that the data is loading data, please try again later, and so on.

Step S103: and inquiring data corresponding to each service ID in the loading cache request queue according to a preset strategy, and loading the inquired data to the cache.

Step S103 specifically includes: according to configured time intervals, service IDs are regularly acquired from a loading cache request queue according to a specific sequence, wherein the number of the service IDs which can be acquired each time is determined through configuration; and inquiring data corresponding to each service ID in the acquired service IDs from the data source.

The specific sequence is specifically according to the enqueuing sequence of the service IDs in the load cache request queue, that is, the service IDs are obtained from the head of the queue each time.

It should be noted that the step numbers of step S102 and step S103 do not indicate the execution sequence, and for each received query request for data, it is checked whether the requested query data is loaded into the cache, and then step S102 is executed, where step S103 is executed periodically, that is, according to the configured time interval, the service ID is acquired periodically and corresponding data is queried from the data source.

In the above embodiment of storing key-value pairs with service IDs as keys by using mapping, after the queried data is loaded into the cache, the key-value pairs corresponding to the service IDs of the queried data in the mapping need to be cleaned.

The embodiment of the invention adopts the scheme of asynchronous loading cache, strives to use the memory resource on the most needed data, does not require the server to immediately return the data to the data inquirer, can avoid the data inquirer from waiting for the data for a long time, and can seek balance in the aspects of providing the data to the data inquirer in time and protecting the data source computing resource.

FIG. 2 is a data loading flow diagram according to one embodiment of the invention.

As shown in fig. 2, the data loading flow of the present embodiment includes steps S201 to S211 as follows.

Step S201: a query request is received.

The query request is used for querying data, wherein the query request includes a service ID of the data requested to be queried, for example, in a scenario of querying commodity evaluation information, the query request includes an evaluation ID requested to be queried, and the evaluation information of a commodity can be queried according to the evaluation ID.

Step S202: checking whether the cache is hit, if yes, executing step S203, otherwise executing step S204.

Whether the cache is hit is checked, that is, whether the data of the requested query is loaded to the cache is checked, if the data is loaded to the cache, the query is requested to hit the cache, otherwise, the query is not hit in the cache.

It should be noted that the data is loaded into the cache, which means that the data exists in the cache and is not expired.

Step S203: and returning the data corresponding to the received query request in the cache.

The data loaded into the cache is arranged into cache objects, and each cache object comprises a service ID and data corresponding to the service ID. And under the condition that the received query request hits the cache, searching data corresponding to the service ID included in the query request in each cache object of the cache, so as to obtain the data corresponding to the received query request.

Step S204: it is determined whether the mapping (denoted as M) already has an equivalent query request of the received query request, if so, step S205 is executed, otherwise, step S206 is executed.

Map (Map) includes key-value pairs, for example in the form:

(key1,value1)

(key2,value2)

......

in each key value pair, key is a service ID, value is a preset flag value, and the preset flag value is set to 1, for example, or other values. The key value pair in the mapping indicates that the cache is being loaded for the service IDs corresponding to the keys, and the service IDs are also stored in a load cache request queue, and the process of loading the cache includes: and inquiring data corresponding to each service ID in the loading cache request queue according to a preset strategy, and loading the inquired data to the cache.

Whether the equivalent query request of the received query request exists in the mapping M is checked, and only whether a key exists in the mapping M and is equal to the service ID included in the query request is checked, if the key exists, the equivalent query request exists, otherwise, the equivalent query request does not exist.

The strategy for checking the equivalent query request of one query request in the embodiment does not need to traverse the whole loading cache request queue, has higher business ID duplicate checking efficiency, and accelerates the duplicate checking speed of the business ID.

Step S205: abandoning the query request and outputting prompt information for the data inquirer.

The prompt information is, for example: "in loading data, please try again later".

In this case, the data inquirer may initiate the inquiry request again to attempt to acquire the data.

Step S206: determining whether the length of the mapping M is greater than a first threshold, if so, performing step S205, otherwise, performing step S207.

Step S207: and adding the key value pair of the received query request to the mapping M by taking the service ID of the received query request as a key and the preset mark value as a value, and adding the service ID of the received query request to the tail of the loading cache request queue.

Step S208: and the background task regularly acquires n service IDs from the head of the loading cache request queue according to a fixed time period t, and arranges the query statements.

The background task is a task for requesting for querying data by the server and can be executed periodically.

The sorting query statement is to generate a query statement for querying data corresponding to the n service IDs, so as to query the data from the data source in the next step.

the specific values of t and n are determined by configuration.

Step S209: and querying data corresponding to the n service IDs from a data source.

That is, the sorted query statement is executed, and the data corresponding to the n service IDs is queried from the data source.

Step S210: and sorting the inquired data into cache objects and filling the cache objects.

Step S211: and clearing the corresponding key value pair in the mapping M according to the service ID added to the cache.

And each service ID added to the Map is added to the loading cache request queue, and when a certain service ID is taken out from the loading cache request queue and data corresponding to the service ID is loaded to a cache, the key value pair corresponding to the service ID in the mapping is cleared so as to ensure that the service IDs in the loading cache request queue and the mapping are the same correspondingly and are the service IDs waiting for loading the cache.

The embodiment of the invention adopts the strategy of asynchronous loading cache, only loads the data required by the data inquirer, strives to save the memory space, avoids the resources of the server side from being rapidly consumed, and can immediately return the corresponding data to the data inquirer for the inquiry request which can hit the cache, and avoids the data inquirer from waiting for a long time.

In addition, the embodiment also performs flow limitation on the query request (flow limitation refers to limiting input flow or output flow of the system, which achieves the purpose of protecting the system), and discards excessive requests, thereby further avoiding that the computing resources of the server are rapidly consumed. In particular, the number of key-value pairs (key-value pairs) in the shot M is dynamically variable. Every time the cache needs to be loaded for a service ID, a key-value pair is added in the mapping M, and every time the cache loading is completed, the mapping M is cleared of the key-value pair. If the rate of adding the key-value pair in the mapping M is greater than the rate of cleaning the key-value pair from the mapping M, the length of the mapping M will continue to increase, if the length of the mapping M increases infinitely, all the memory of the server will be consumed, and meanwhile, too long length of the mapping M also means that the number of query requests is too large, so that the computing resources of the server are consumed rapidly. Therefore, the embodiment sets a first threshold, determines whether the length of the mapping M is greater than the first threshold before adding the key-value pair to the mapping M each time, and discards the received query request instead of adding the key-value pair to the mapping M when the length of the mapping M is greater than the first threshold, thereby implementing the current limitation on the query request and reducing the consumption of the computing resources of the server.

Finally, the background task for querying data regularly acquires n service IDs from the head of the load cache request queue according to a fixed time period t, sorts query statements and queries corresponding data, and the interval time t for executing the background task and the number n of the service IDs corresponding to the queried data are determined by configuration, so that the rate of acquiring data from a data source is controlled, and sudden high concurrent access to the data source is avoided.

FIG. 3 is a schematic diagram illustrating the loading of data according to another embodiment of the invention.

As shown in fig. 3, the principle of data loading of the present embodiment is explained as follows:

each time the server receives a request for querying data (i.e., a query request for short), it checks whether the cache can be hit, and if so, returns the data in the cache, as shown in step 1 of fig. 3.

If not, checking whether there is an equivalent query request of the query request, and step 2 in fig. 3 shows a mapping M, where for each key-value pair, key is a service ID, and value is a preset flag value, indicating that the system (referred to as the server) is loading the cache for the service ID. It is checked whether the system is caching for a load of a certain service ID, as long as it checks whether there is a key in the mapping M equal to this service ID. In the case that there is no key equal to the service ID included in the query request, the service ID of the query request is appended to the Map, specifically, the service ID of the query request is taken as a key, a flag value, for example, 1, is taken as a value, a key value pair is generated and added to the Map M, so as to ensure that the service ID appears in the Map, as shown in step 2 in fig. 3.

If the equivalent query request does not exist, after the service ID of the query request is appended to the Map, the service ID needs to be appended to the tail of the load cache request Queue (Queue), as shown in step 3 in fig. 3.

If there is the above equivalent query request, which indicates that the system is already loading the cache for this service ID, the query request is discarded, and the data inquirer is prompted that the system is loading data, please try again later.

In the mapping M, the number of key-value pairs is dynamically changed, each time a cache needs to be loaded for one service ID, one key-value pair is added in the mapping M, and each time the cache loading is completed, the mapping M is cleared of the key-value pairs. In order to avoid the too fast increase of the length of the mapping M, the method of the present invention is to check the length of the mapping M before appending the key-value pair to the mapping M each time, and if the length is greater than the threshold V1, no key-value is appended to the mapping M any more, and the data inquirer is directly prompted to load the data and try again later. When the data inquirer initiates the request again, the server may accept the request.

The cache loading is completed by a background task, the background task is executed periodically, n (configurable) service IDs are obtained from the head of the load cache request queue every time period t (configurable), and query statements are arranged to query data in the data source, as shown in step 4 in fig. 3.

Then, the cache objects are sorted according to the inquired data, and are filled in the memory cache, and the expiration time is specified, as shown in step 5 of fig. 3.

After the memory cache is successfully filled with a certain service ID, the corresponding key-value pair in the mapping M needs to be cleared, which indicates that the loading cache is completed, as shown in step 6 in fig. 3.

In combination with monitoring, the embodiment of the invention adjusts t and n in time, which not only can ensure that the resources of the server are not consumed quickly, but also can ensure that the cache is loaded and completed as soon as possible, and provide data to a data inquirer as soon as possible.

FIG. 4 is a schematic diagram of a data loading flow according to another embodiment of the invention.

The data loading flow of the present embodiment includes steps S401 to S410 as follows.

Step S401: a query request is received.

Step S402: checking whether the cache is hit, if yes, executing step S403, otherwise executing step S404.

Step S403: and returning the data corresponding to the received query request in the cache.

Step S404: and judging whether the equivalent query request of the received query request exists in the load cache request queue, if so, executing the step S405, otherwise, executing the step S406.

Step S405: abandoning the query request and outputting prompt information for the data inquirer.

Step S406: and judging whether the length of the load cache request queue is greater than a second threshold value, if so, executing the step S405, otherwise, executing the step S407.

Step S407: and adding the service ID of the received query request to the tail of the loading cache request queue.

Step S408: and the background task regularly acquires n service IDs from the head of the loading cache request queue according to a fixed time period t, and arranges the query statements.

Step S409: and querying data corresponding to the n service IDs from a data source.

Step S410: and sorting the inquired data into cache objects and filling the cache objects.

The above steps S401 to S403 and S405 refer to the descriptions of steps S201 to S203 and S205. The steps S408 to S410 are specifically described with reference to the steps S208 to S210.

The cache is asynchronously loaded, but not synchronously loaded, by the background task, so that a data inquirer is prevented from waiting for a long time, the rate of acquiring data from a data source can be controlled, sudden high concurrent access to the data source is avoided, the inquiry request can be limited, and excessive requests are discarded, so that server resources are prevented from being seriously consumed.

Fig. 5 is a schematic diagram of main modules of a data loading apparatus according to an embodiment of the present invention.

As shown in fig. 5, the data loading apparatus 500 according to the embodiment of the present invention mainly includes: the system comprises a data cache checking module 501, a cache data returning module 502, a query request processing module 503 and a data query and loading module 504.

A data cache checking module 501, configured to check whether data requested to be queried is loaded into a cache after each query request for the data is received, where the query request includes a service ID of the data requested to be queried.

A cache data returning module 502, configured to return the cached data requested to be queried if the data requested to be queried is already loaded into the cache.

A query request processing module 503, configured to check whether an equivalent query request of the query request already exists if the data requested to be queried is not loaded to the cache, where the equivalent query request and the query request include the same service ID; if yes, abandoning the query request; otherwise, the service ID included in the query request is added to the load cache request queue.

In one embodiment:

the query request processing module 503 may include a first equivalent query request checking sub-module for: and checking whether a key which is equal to the service ID included by the query request exists in the mapping, if so, then, existing the equivalent query request of the query request, otherwise, not, existing the equivalent query request, wherein the mapping includes key-value pairs, the key of the key-value pair is the service ID, and the value is a preset mark value.

The query request processing module may be specifically configured to: and generating a key-value pair by taking the service ID included in the inquiry request as a key and a preset mark value as a value when the key equal to the service ID included in the inquiry request does not exist in the mapping, and adding the generated key-value pair to the mapping.

Before adding the service ID included in the query request to the load cache request queue and adding the generated key value pair to the mapping, the query request processing module 503 may further check whether the length of the mapping is greater than a first threshold, perform an operation of adding the service ID included in the query request to the load cache request queue under the condition that it is determined that the length of the mapping is greater than the first threshold, generate a key value pair with the service ID included in the query request as a key and a preset flag value as a value, and add the generated key value pair to the mapping.

If the length of the mapping is less than or equal to the first threshold, the query request is discarded.

In another embodiment:

query request processing module 503 may include a second equivalent query request checking sub-module for: and traversing the load cache request queue to check whether the load cache request queue stores the service ID included by the query request, if so, existing the equivalent query request of the query request, and otherwise, not existing the equivalent query request.

Before adding the service ID included in the query request to the load cache request queue, the query request processing module 503 may check whether the length of the load cache request queue is greater than a second threshold, and perform an operation of adding the service ID included in the query request to the load cache request queue on the condition that it is determined that the length of the load cache request queue is greater than the second threshold.

If the length of the load cache request queue is less than or equal to the second threshold, the query request is discarded.

The data query and load module 504 is configured to query, according to a preset policy, data corresponding to each service ID in the load cache request queue, and load the queried data into the cache.

The data query and load module 504 specifically includes a data query sub-module, configured to: according to configured time intervals, service IDs are regularly acquired from a loading cache request queue according to a specific sequence, wherein the number of the service IDs which can be acquired each time is determined through configuration; and inquiring data corresponding to each service ID in the acquired service IDs from the data source.

In the above embodiment of storing key-value pairs keyed by service IDs using mapping, the data loading apparatus 500 may further include a cleaning module, configured to: and after the inquired data is loaded into the cache, clearing the corresponding key value pair of the service ID of the inquired data in the mapping.

In addition, the specific implementation of the data loading apparatus in the embodiment of the present invention has been described in detail in the above data loading method, and therefore, the repeated content is not described again.

Fig. 6 shows an exemplary system architecture 600 to which the data loading method or the data loading apparatus according to the embodiment of the present invention may be applied.

As shown in fig. 6, the system architecture 600 may include

terminal devices

601, 602, 603, a network 604, and a server 605. The network 604 serves to provide a medium for communication links between the

terminal devices

601, 602, 603 and the server 605. Network 604 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.

A user may use the

terminal devices

601, 602, 603 to interact with the server 605 via the network 604 to receive or send messages or the like. The

terminal devices

601, 602, 603 may have installed thereon various communication client applications, such as shopping applications, web browser applications, search applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).

The

terminal devices

601, 602, 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 605 may be a server providing various services, such as a background management server (for example only) providing support for shopping websites browsed by users using the

terminal devices

601, 602, 603. The backend management server may analyze and perform other processing on the received data such as the product information query request, and feed back a processing result (for example, product information — just an example) to the terminal device.

It should be noted that the data loading method provided by the embodiment of the present invention is generally executed by the server 605, and accordingly, the data loading apparatus is generally disposed in the server 605.

It should be understood that the number of terminal devices, networks, and servers in fig. 6 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Referring now to FIG. 7, shown is a block diagram of a computer system 700 suitable for use in implementing a terminal device or server of an embodiment of the present application. The terminal device or the server shown in fig. 7 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.

As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU)701, which can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data necessary for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.

The following components are connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read out therefrom is mounted into the storage section 708 as necessary.

In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 709, and/or installed from the removable medium 711. The computer program executes the above-described functions defined in the system of the present application when executed by the Central Processing Unit (CPU) 701.

It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor comprises a data cache checking module, a cache data returning module, a query request processing module and a data query and loading module. Where the names of these modules do not constitute a limitation on the module itself under certain circumstances, for example, the data cache check module may also be described as "a module for checking, after each query request for data is received, whether the data requesting the query has been loaded into the cache".

As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: after receiving a query request for data, checking whether the data requested to be queried is loaded to a cache or not, wherein the query request comprises a service ID of the data requested to be queried; if the data requested to be queried is loaded to the cache, returning the cached data requested to be queried; if the data requested to be queried is not loaded to the cache, checking whether an equivalent query request of the query request already exists, wherein the equivalent query request and the query request comprise the same service ID; if so, discarding the query request; otherwise, adding the service ID included in the query request to a loading cache request queue; and querying data corresponding to each service ID in the load cache request queue according to a preset strategy, and loading the queried data to the cache.

According to the technical scheme of the embodiment of the invention, after receiving the query request for the data, checking whether the data requested to be queried is loaded to the cache or not, and if the data requested to be queried is loaded to the cache, returning the cached data requested to be queried; if the data requested to be inquired is not loaded in the cache, checking whether an equivalent inquiry request of the inquiry request exists, if so, abandoning the inquiry request, otherwise, adding the service ID included in the inquiry request into a loading cache request queue, inquiring the data corresponding to each service ID in the loading cache request queue according to a preset strategy, and loading the inquired data into the cache. The method and the device can avoid long-time waiting of the data inquirer, only need to load the data required by the data inquirer, and save the memory space. The length of mapping or the length of a loading cache request queue is controlled by a threshold value, so that the current limitation of the query request is realized, excessive requests are discarded, the computing resources of the server are prevented from being rapidly consumed, and the server is ensured to continuously provide services. According to the configured time interval, service IDs are regularly acquired from the load cache request queue according to a specific sequence, and the quantity of the service IDs which can be acquired each time is determined through configuration, so that the data acquisition rate from the data source query can be controlled, instantaneous high concurrent access to the data source is avoided, the computing resources of the data source are prevented from being rapidly consumed, and the data source is ensured to continuously provide services.

The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for loading data, comprising:

after receiving a query request for data, checking whether the data requested to be queried is loaded to a cache or not, wherein the query request comprises a service ID of the data requested to be queried;

if the data requested to be queried is loaded to the cache, returning the cached data requested to be queried;

if the data requested to be queried is not loaded to the cache, checking whether an equivalent query request of the query request already exists, wherein the equivalent query request and the query request comprise the same service ID; if so, discarding the query request; otherwise, adding the service ID included in the query request to a loading cache request queue;

and querying data corresponding to each service ID in the load cache request queue according to a preset strategy, and loading the queried data to the cache.

2. The method of claim 1, wherein the step of checking whether there is an equivalent query request to the query request comprises:

and checking whether a key which is equal to the service ID included in the query request exists in a mapping, if so, the equivalent query request exists, otherwise, the equivalent query request does not exist, wherein the mapping includes a key-value pair, the key of the key-value pair is the service ID, and the value is a preset mark value.

3. The method of claim 2, further comprising: in the absence of a key in the mapping equal to the service ID included in the query request,

and generating a key value pair by taking the service ID included in the query request as a key and the preset mark value as a value, and adding the generated key value pair to the mapping.

4. The method according to claim 3, wherein the operation of adding the service ID included in the query request to the load cache request queue is performed on the condition that it is checked and determined that the length of the mapping is greater than the first threshold, and the operation of generating a key-value pair by using the service ID included in the query request as a key and using the preset flag value as a value, and adding the generated key-value pair to the mapping is performed.

5. The method according to claim 2, wherein after the queried data is loaded into the cache, the corresponding key-value pairs of the service IDs of the queried data in the mapping are cleared.

6. The method of claim 1, wherein the step of checking whether there is an equivalent query request to the query request comprises:

and traversing the loading cache request queue to check whether the loading cache request queue stores the service ID included by the query request, if so, the equivalent query request exists, otherwise, the equivalent query request does not exist.

7. The method as claimed in claim 6, wherein the operation of adding the service ID included in the query request to the load cache request queue is performed on condition that the length of the load cache request queue is checked and determined to be greater than the second threshold.

8. The method according to claim 1, wherein the step of querying the data corresponding to each service ID in the load cache request queue according to a preset policy comprises:

according to configured time intervals, service IDs are regularly acquired from the load cache request queue according to a specific sequence, wherein the number of the service IDs which can be acquired each time is determined through configuration;

and inquiring data corresponding to each service ID in the acquired service IDs from the data source.

9. A data loading apparatus, comprising:

the data cache checking module is used for checking whether the data requested to be inquired is loaded to a cache after receiving an inquiry request for the data, wherein the inquiry request comprises a service ID of the data requested to be inquired;

the cache data returning module is used for returning the cached data requested to be queried if the data requested to be queried is loaded to the cache;

a query request processing module, configured to check whether an equivalent query request of the query request already exists if the data requested to be queried is not loaded to the cache, where the equivalent query request and the query request include the same service ID; if so, discarding the query request; otherwise, adding the service ID included in the query request to a loading cache request queue;

and the data query and loading module is used for querying data corresponding to each service ID in the loading cache request queue according to a preset strategy and loading the queried data to the cache.

10. The apparatus of claim 9, wherein the query request processing module comprises a first equivalent query request checking sub-module configured to:

11. The apparatus of claim 10, wherein the query request processing module is further configured to:

and under the condition that a key equal to the service ID included in the query request does not exist in the mapping, generating a key-value pair by taking the service ID included in the query request as the key and the preset mark value as the value, and adding the generated key-value pair to the mapping.

12. The apparatus according to claim 11, wherein the query request processing module performs the operation of adding the service ID included in the query request to a load cache request queue on a condition that it is checked and determined that the length of the mapping is greater than a first threshold, and the operation of generating a key-value pair by using the service ID included in the query request as a key and using the preset flag value as a value, and adding the generated key-value pair to the mapping.

13. The apparatus of claim 10, further comprising a cleaning module to:

and after the inquired data is loaded into the cache, cleaning the corresponding key value pair of the service ID of the inquired data in the mapping.

14. The apparatus of claim 9, wherein the query request processing module comprises a second equivalent query request checking sub-module configured to:

15. The apparatus as claimed in claim 14, wherein the query request processing module performs the operation of adding the service ID included in the query request to the load cache request queue on condition that the length of the load cache request queue is checked and determined to be greater than the second threshold.

16. The apparatus of claim 9, wherein the data query and load module comprises a data query submodule configured to:

17. An electronic device, comprising:

one or more processors;

a memory for storing one or more programs,

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-8.

18. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-8.