CN115221438A

CN115221438A - Data access method, apparatus, device, medium and product

Info

Publication number: CN115221438A
Application number: CN202210812674.6A
Authority: CN
Inventors: 李冰; 林振溪; 徐祎; 屠方轫
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2022-07-12
Filing date: 2022-07-12
Publication date: 2022-10-21

Abstract

The present application relates to a data access method, apparatus, device, medium and product. The method comprises the following steps: receiving a calling request sent by a back end according to a data access request; the data access request is received by a proxy end from a front-end page, and the data access request is sent to the back end after the proxy end is determined not to store first data corresponding to the data access request; and under the condition that the first data are determined to be stored according to the calling request, feeding back the first data to the back end so as to feed back the first data to the front-end page through the back end. The method can meet the high concurrent access requirement.

Description

Data access method, apparatus, device, medium and product

Technical Field

The present application relates to the field of computer technologies, and in particular, to a data access method, apparatus, device, medium, and product.

Background

With the explosive growth of internet technology and the explosive growth of information, databases often store large amounts of data. For example, data generated in an enterprise site, data entered by a user accessing an enterprise site, and data introduced by an enterprise.

Under the scene of high concurrent access, a large number of users access the database at the same time, so that great access pressure is caused to the database, and the database is easy to crash. Therefore, how to deal with the high concurrent access requirement becomes a technical problem to be solved urgently.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a data access method, apparatus, device, medium, and product that can cope with high concurrent access requirements in view of the above technical problems.

In a first aspect, the present application provides a data access method, which is applied to an object storage side, and the method includes:

receiving a calling request sent by a back end according to a data access request; the data access request is received by the agent terminal from a front-end page, and the agent terminal is sent to the back-end after determining that the agent terminal does not store first data corresponding to the data access request;

and under the condition that the first data are determined to be stored according to the calling request, feeding back the first data to the back end so as to feed back the first data to the front-end page through the back end.

In one embodiment, the feeding back the first data to the front end page through the back end includes:

and under the condition that the first data is in the first data format, the first data is sent to the agent end through the back end for storage, and the first data is fed back to the front end page through the agent end.

In one embodiment, the data access method further includes:

and receiving second data sent by the front-end page according to a preset period, and storing the second data.

In one embodiment, the feeding back the first data to the front page through the back end includes:

and under the condition that the first data is in the second data format and is not empty, feeding back the first data to the front-end page through the back end.

In one embodiment, the data access method further includes:

storing the third data sent by the cache end; and the third data is obtained from the database and sent to the cache terminal for storage according to the data access request under the condition that the back terminal determines that the first data is in the second data format and is empty.

In one embodiment, the data access method further includes:

storing fourth data sent by a cache end; the fourth data is sent to the cache end for storage after the back end receives the data change request sent by the front end page.

In one embodiment, the data access method further includes:

and if the data access request does not match the preset database access condition, discarding the data access request.

In one embodiment, the data access method further includes:

if a plurality of requests for accessing the same data are received, acquiring the data according to a first request with the earliest access time, and sending the acquired data to a cache end for storage;

and acquiring data corresponding to a second request except the first request in the multiple data access requests from the cache terminal.

In one embodiment, the data access method further includes:

and if the data in the object storage terminal is out of date, inquiring the data from the database and updating the object storage terminal.

The application provides a data access method, which is applied to a back end and comprises the following steps:

receiving a data access request sent by an agent terminal; the data access request is received by the agent terminal from a front-end page and is sent after the agent terminal is determined not to store first data corresponding to the data access request;

calling an object storage end according to the data access request;

and under the condition that the object storage end is determined to store the first data, feeding the first data back to the front-end page.

In one embodiment, the feeding back the first data to the front page includes:

and under the condition that the first data is in the first data format, sending the first data to the agent end for storage, and feeding the first data back to the front-end page through the agent end.

In one embodiment, the feeding back the first data to the front end page further includes:

and feeding back the first data to the front end page under the condition that the first data is in the second data format and is not empty.

In one embodiment, the data access method further includes:

under the condition that the first data is in a second data format and is empty, acquiring third data from the database according to the data access request;

and feeding back the third data to the front-end page through the proxy end, and sending the first data to the cache end for storage.

In one embodiment, the data access method further includes:

and receiving a data change request sent by the front-end page, and respectively sending fourth data corresponding to the data change request to the database and the cache end for storage.

In a second aspect, the present application further provides a data access apparatus, applied to an object storage end, where the apparatus includes:

the calling request receiving module is used for receiving a calling request sent by a back end according to the data access request; the data access request is received by the agent terminal from a front-end page, and the agent terminal is sent to the back-end after determining that the agent terminal does not store first data corresponding to the data access request;

and the data feedback module is used for feeding back the first data to the back end under the condition that the first data are determined to be stored according to the calling request, so that the first data are fed back to the front-end page through the back end.

In one embodiment, the data feedback module is specifically configured to send the first data to the agent for storage through the backend and feed the first data back to the front-end page through the agent when the first data is in the first data format.

In one embodiment, the data feedback module is specifically configured to receive second data sent by the front-end page according to a preset period, and store the second data.

In one embodiment, the data feedback module is specifically configured to feed back the first data to the front-end page through the back end when the first data is in the second data format and is not empty.

In one embodiment, the apparatus further comprises:

the data storage module is used for storing third data sent by the cache end; and the third data is obtained from the database and sent to the cache terminal for storage according to the data access request under the condition that the back end determines that the first data is in the second data format and is empty.

In one embodiment, the data storage module is further configured to store fourth data sent by the cache terminal; the fourth data is sent to the cache end for storage after the back end receives the data change request sent by the front end page.

In one embodiment, the apparatus further comprises:

and the discarding module is used for discarding the data access request if the data access request is not matched with the preset database access condition.

In one embodiment, the apparatus further comprises:

the first data acquisition module is used for acquiring data according to a first request with the earliest access time and sending the acquired data to a cache end for storage if a plurality of requests for accessing the same data are received;

and the second data acquisition module is used for acquiring data corresponding to a second request except the first request in the multiple data access requests from the cache terminal.

In one embodiment, the apparatus further comprises:

and the updating module is used for inquiring data from the database and updating the object storage end if the data in the object storage end is out of date.

The present application further provides a data access device, which is applied to a backend, and the device includes:

the request receiving module is used for receiving a data access request sent by the agent terminal; the data access request is received by the agent terminal from a front-end page and is sent after the agent terminal is determined not to store first data corresponding to the data access request;

the calling module is used for calling the object storage end according to the data access request;

and the data feedback module is used for feeding back the first data to the front-end page under the condition that the object storage end is determined to store the first data.

In one embodiment, the data feedback module is specifically configured to send the first data to the agent for storage when the first data is in the first data format, and feed the first data back to the front-end page through the agent.

In one embodiment, the data feedback module is specifically configured to feed back the first data to the front page when the first data is in the second data format and is not empty.

In one embodiment, the apparatus further comprises:

the data acquisition module is used for acquiring third data from the database according to the data access request under the condition that the first data is in the second data format and is empty;

and the data storage module is used for feeding back the third data to the front-end page through the proxy end and sending the first data to the cache end for storage.

In one embodiment, the data storage module is further configured to receive a data change request sent by a front-end page, and send fourth data corresponding to the data change request to the database and the cache end for storage, respectively.

In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the steps of the first aspect when executing the computer program.

In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the first aspect.

In a fifth aspect, the present application further provides a computer program product. The computer program product comprising a computer program that when executed by a processor performs the steps of the first aspect.

According to the data access method, device, equipment, medium and product, the object storage end receives the call request sent by the back end according to the data access request; the data access request is received by the agent terminal from a front-end page, and the agent terminal is sent to the back-end after determining that the agent terminal does not store first data corresponding to the data access request; and under the condition that the first data are determined to be stored according to the calling request, feeding back the first data to the back end so as to feed back the first data to the front end page through the back end. In the embodiment of the application, the proxy terminal and the object storage terminal are adopted for data storage, and when the front-end page sends a data access request, first data corresponding to the data access request can be searched for at the proxy terminal; under the condition that the agent end does not store the first data, searching the first data from the object storage end; and under the condition that the object storage end stores the first data, feeding the first data back to the front-end page through the back end. Therefore, under the scene of high concurrent access, a large number of data access requests can access the agent end and the object storage end in sequence, so that the condition that a large number of data access requests access the database at the same time and great access pressure is caused to the database is avoided, high concurrent access requirements can be met, and the problem that the database crashes due to overlarge access pressure is avoided.

Drawings

FIG. 1 is a diagram of an application environment of a data access method in one embodiment;

FIG. 2 is one of the flow diagrams of a data access method in one embodiment;

FIG. 3 is a second flowchart of a data access method according to an embodiment;

FIG. 4 is a third flowchart illustrating a data access method according to an embodiment;

FIG. 5 is a fourth flowchart illustrating a data access method according to an embodiment;

FIG. 6 is a fifth flowchart of a data access method in one embodiment;

FIG. 7 is a block diagram showing the structure of a data access device according to one embodiment;

FIG. 8 is a second block diagram illustrating the structure of a data access device according to an embodiment;

FIG. 9 is a block diagram of a third example of a data access device;

FIG. 10 is a block diagram showing the structure of a data access device according to an embodiment;

FIG. 11 is a block diagram showing the structure of a data access device according to the first embodiment;

FIG. 12 is a sixth block diagram showing the structure of a data access device according to the first embodiment;

FIG. 13 is a seventh block diagram showing the configuration of a data access device in one embodiment;

FIG. 14 is a diagram illustrating an internal structure of a computer device according to an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

The data access method provided by the embodiment of the application can be applied to the application environment shown in fig. 1. The application environment comprises a front end 01, an agent end 02, a back end 03, an object storage end 04, a database 05 and a cache end 06, wherein the front end 01 can communicate with the agent end 02 and the back end 03, the agent end 02 can also communicate with the back end 03, the object storage end 04 and the cache end 06, the back end 03 can also communicate with the object storage end 04, the cache end 06 and the database 05, and the object storage end 04 can also communicate with the database 05. The front end 01, the agent end 02, the back end 03, the object storage end 04, the database 05 and the cache end 06 may be arranged on the same server or on different servers. For example, the user may set the front end 01, the proxy end 02, and the back end 03 on the same server, set the object storage end 04 and the cache end 06 on the same server, and set the database 05 on one server separately.

In an embodiment, fig. 2 is a flowchart of a data access method, which is described by taking an example that the method is applied to an object storage end, and may include the following steps:

and S101, receiving a calling request sent by a back end according to the data access request.

The data access request is received by the agent terminal from the front-end page, and the agent terminal is sent to the back-end after determining that the first data corresponding to the data access request is not stored in the agent terminal.

The proxy terminal refers to a high-performance Http and reverse proxy server, uses a reverse proxy mode, and is characterized by small occupied memory and strong concurrency capability. The reverse proxy mode is that a proxy server receives a connection request on the Internet, then forwards the request to a server on an internal network, and returns a result obtained from the server to a client requesting connection on the Internet, and at the moment, the proxy server externally shows a reverse proxy server.

The front page sends a data access request to the agent side; the proxy end receives the data access request, and sends the data access request to the back end after determining that the first data corresponding to the data access request is not stored; and the back end sends a calling request to the object storage end according to the data access request. And the object storage end receives a calling request sent by the back end.

And S102, under the condition that the first data are determined to be stored according to the calling request, feeding the first data back to the rear end so as to feed the first data back to the front end page through the rear end.

Wherein the first data may comprise a static page and JSON (JavaScript Object notification) data.

And the object storage end determines whether the first data are stored or not according to the calling request, and feeds the first data back to the back end under the condition that the first data are determined to be stored. The back-end receives the first data and forwards it to the front-end page.

In the above embodiment, the object storage end receives a call request sent by the back end according to the data access request; the data access request is received by the agent terminal from a front-end page, and the agent terminal is sent to the back-end after determining that the agent terminal does not store first data corresponding to the data access request; and under the condition that the first data are determined to be stored according to the calling request, feeding back the first data to the back end so as to feed back the first data to the front-end page through the back end. In the embodiment of the application, the agent terminal and the object storage terminal are adopted for data storage, and when a data access request is sent by a front-end page, first data corresponding to the data access request can be searched for by the agent terminal; under the condition that the agent end does not store the first data, searching the first data from the object storage end; and under the condition that the object storage end stores the first data, feeding the first data back to the front-end page through the back end. Therefore, under the scene of high concurrent access, a large number of data access requests can access the agent end and the object storage end in sequence, so that the condition that a large number of data access requests access the database at the same time and great access pressure is caused to the database is avoided, high concurrent access requirements can be met, and the problem that the database crashes due to overlarge access pressure is avoided.

In an embodiment, the step of feeding back the first data to the front page through the back end may include: and under the condition that the first data is in the first data format, the first data is sent to the agent end through the back end for storage, and the first data is fed back to the front end page through the agent end.

The first data format refers to a static page, the static page exists actually, the static page is directly loaded to a client browser to be displayed without compiling of a server, and common static pages include html extension and html extension. The embodiment of the application does not limit the static page.

The object storage end sends the static page to the back end; the back end sends the static page to the agent end; the proxy side stores the static page and forwards the static page to the front side; and the front end displays the page after receiving the static page.

In the embodiment, when the first data is in the first data format, the first data is sent to the proxy end through the back end to be stored, and the first data is fed back to the front-end page through the proxy end.

In an embodiment, the step of feeding back the first data to the front page through the back end may further include: and receiving second data sent by the front-end page according to a preset period, and storing the second data.

The second data refers to an updated static page, information is frequently updated due to the fact that the static page is a page frequently visited by a user, manual collection is performed manually, information is input in batches through an external website, and the updating speed of the static page is one time every five minutes.

The cache terminal sends a call request to the front-end page every five minutes through the HttpClient, the front-end page receives the call request, and the static page of the cache terminal is updated according to the call request; and the front-end page sends the updated static page to the object storage end for storage.

The HTTP client is a child under Apache Jakarta Common to provide an efficient, up-to-date, feature-rich client programming toolkit supporting the HTTP protocol, and it supports the latest versions and recommendations of the HTTP protocol.

In the embodiment, due to the fact that the static page access amount is high and the data are updated frequently, the object storage end receives the second data sent by the front-end page according to the preset period and stores the second data, the regular updating and the regular storage of the static page are achieved, and the regular updating of the static page enables a user to obtain related hot spot information in time; the static page is stored into the object storage end at regular time, so that the influence of a large number of data access requests on accessing the database simultaneously on the performance of the database in a high concurrent access scene can be reduced.

In an embodiment, the step of feeding back the first data to the front page through the back end may further include: and under the condition that the first data is in the second data format and is not empty, feeding back the first data to the front-end page through the back end.

The second data format refers to a JSON data format, and JSON is a lightweight data exchange format. JSON employs a text format that is completely language independent, and these features make JSON an ideal data exchange language that is easy for humans to read and write, and also easy for machine parsing and generation.

The object storage end sends the first data to the back end; the back end determines that the first data is JSON data and determines whether the JSON data is null or not; and under the condition that the JSON data is not empty, the back end feeds back the JSON data to the front end page.

In the above embodiment, when the first data is in the second data format and is not empty, the first data is fed back to the front-end page through the back end, so that the front-end page obtains corresponding JSON data, and the JSON data is displayed to the user; the first data are sent to the back end through the object storage end, and the problems that reading of the first data in the database is slow and performance consumption is large in a high concurrency scene are solved.

On the basis of the above embodiments, the embodiments of the present application may further include the following steps: the object storage end stores the third data sent by the cache end; and the third data is obtained from the database and sent to the cache terminal for storage according to the data access request under the condition that the back terminal determines that the first data is in the second data format and is empty.

The type of the database can be MySQL, mariaDB, percona Server, postgreSQL and the like, and the type of the database is not limited in the embodiment of the application and can be specifically set according to actual conditions. Kafka in the cache end is an open source stream processing platform, and has three transmission modes of synchronous (sync), asynchronous (async) and oneway, and the embodiment of the application adopts an asynchronous transmission mode. When using the asynchronous sending mode, the caller can process other tasks without waiting for a response after sending a message, unlike the synchronous mode, in which two application systems (the sending program and the receiving program) do not have to be running at the same time, nor do they have to process communication tasks at the same time.

The object store sends the first data to the backend. The back end determines that the first data is JSON data and determines whether the JSON data is null or not; and under the condition that the JSON data is empty, acquiring third data from the database according to the data access request. And the back end asynchronously sends the third data to the cache end, a Kafka producer in the cache end can monitor in real time after receiving the third data, the Kafka consumer consumes the third data monitored in real time by the Kafka producer, and further, the cache end sends the third data to the object storage end for storage.

In the above embodiment, the object storage end stores the third data sent by the cache end; and the third data is obtained from the database and sent to the cache terminal for storage according to the data access request under the condition that the back terminal determines that the first data is in the second data format and is empty. According to the embodiment of the application, the object storage end stores the third data sent by the cache end, so that the third data can be directly obtained from the object storage end when a user accesses the third data next time, the third data does not need to be obtained from the database according to the data access request again, convenience and high efficiency are achieved, the obtaining efficiency of the third data is improved, and the pressure of the database is reduced.

On the basis of the above embodiments, the embodiments of the present application may further include the following steps: storing fourth data sent by a cache end; the fourth data is sent to the cache end for storage after the back end receives the data change request sent by the front end page.

Wherein the fourth data refers to an updated detailed page. The cache terminal refers to a Remote directory Server (Redis), which is a type of memory cache Server.

The front-end page sends a data change request to the back end; the back end receives a data change request sent by the front end page and asynchronously sends fourth data to the cache end; and after receiving the fourth data, the Kafka producer in the cache terminal can monitor the fourth data in real time, the Kafka consumer consumes the fourth data monitored by the Kafka producer in real time, and further, the cache terminal sends the fourth data to the object storage terminal for storage.

In the above embodiment, the fourth data refers to an updated detailed page, and the object storage end stores the fourth data sent by the cache end; the fourth data is sent to the cache end for storage after the back end receives the data change request sent by the front end page. According to the embodiment of the application, the change and the storage of the detailed page are realized, so that a user can obtain the changed detailed page from the object storage end, and the time is saved.

On the basis of the above embodiments, the embodiments of the present application may further include the following steps: and if the data access request does not match the preset database access condition, discarding the data access request.

The preset database access condition is set in the object storage end, and may be an ID number of a user or an IP address of a user terminal device.

For example, an anonymous user logs in a data access system, sends a data access request to perform data access, an object storage terminal searches for an ID number of the anonymous user corresponding to the data access request, and if the ID number cannot be found, the data access request is discarded. For another example, the anonymous user logs in the data access system, sends a data access request to perform data access, the target storage end searches the IP address of the anonymous user terminal device corresponding to the data access request, and if the IP address cannot be searched, the data access request is discarded.

In the above embodiment, since the preset database access condition is set in the object storage, when the anonymous user continuously initiates the data access request, the data access request is matched with the preset database access condition in the object storage, and if the data access request is not matched with the preset database access condition, the object storage discards the data access request; only when the data access request is matched with the preset database access condition, the object storage end allows the data access request to access the database, and further the problem that the database pressure is too high when an anonymous user continuously initiates the data access request is solved.

In an embodiment, fig. 3 is a flowchart of a data access method, which is described by taking an example that the method is applied to an object storage side, and may further include the following steps:

s201, if a plurality of requests for accessing the same data are received, acquiring the data according to the first request with the earliest access time, and sending the acquired data to a cache end for storage.

The same data is generally hot data, some hot data is not in the object storage end, but in the database, when the object storage end receives a plurality of requests for accessing the hot data, due to the fact that the concurrency is particularly large, the data cannot be read in the object storage end, all the requests can be synchronously read out of the database, and the pressure of the database is increased instantly. In order to avoid the problem, when the object storage end receives a plurality of requests for accessing the same data, the object storage end acquires the data from the database according to the first request with the earliest access time, and sends the acquired data to the cache end for storage.

For example, the object storage end receives a data access request with an access time of 8, a data access request with an access time of 9.

S202, data corresponding to a second request except the first request in the multiple data access requests are obtained from the cache end.

And the data corresponding to the second request except the first request is the same as the data accessed by the first request.

Because the first request and the second request except the first request access the same data, after the cache end stores the data acquired by the first request, the second request except the first request acquires the corresponding data from the cache end.

In the embodiment, if a plurality of requests for accessing the same data are received, the data are acquired according to the first request with the earliest access time, so that the problem of overlarge pressure of the database when the database is accessed simultaneously is solved; the data corresponding to the second request except the first request in the multiple data access requests are obtained from the cache end, so that the problem that the performance of the database is affected when the second request except the first request obtains the data from the database again is solved; and the cache end is used for effectively resisting instantaneous surge traffic, so that important references are provided for the subsequent reconstruction of other websites and the conditions of similar scenes.

On the basis of the above embodiments, the embodiments of the present application may further include the following steps: and if the data in the object storage terminal is out of date, inquiring the data from the database and updating the object storage terminal.

The reason why the expiration time is randomly set for the data in the object storage end is that when a hacker maliciously attacks the data access system or the user information is empty, whether the data exist in the object storage end is firstly inquired, but when a large amount of data in the object storage end is expired at the same time, all the data are inquired from the database, so that the database is over-stressed and even crashes.

If the data in the object storage terminal is overdue, inquiring the data from the database; if the data is found, updating the data of the object storage end; if not, return null.

In the embodiment, as the data in the object storage end is randomly provided with the expiration time, the phenomenon that a large amount of data expires at the same time can be prevented; the expired data can be inquired and obtained from the database, and the unexpired data can be continuously obtained from the object storage terminal, so that the problem that when a large amount of data is expired simultaneously, all data is inquired from the database, and the database is over-stressed or even down is solved.

In one embodiment, as shown in fig. 4, a data access method is provided, which is described by taking the method as an example for being applied to a backend, and may include the following steps:

s301, receiving a data access request sent by an agent terminal.

The data access request is received by the agent terminal from the front-end page and is sent after the agent terminal is determined not to store first data corresponding to the data access request.

The front page sends a data access request to the agent side; the proxy end receives the data access request, and sends the data access request to the back end after determining that the first data corresponding to the data access request is not stored; and the back end receives the data access request sent by the agent end.

S302, calling an object storage terminal according to the data access request.

The object storage end is an object storage service platform which is built by itself.

And the back end calls the object storage end according to the data access request.

And S303, feeding back the first data to the front-end page under the condition that the object storage end is determined to store the first data.

The object storage end receives the data access request, and returns the first data to the back end under the condition that the first data are determined to be stored; the back end returns the first data to the agent end, and the agent end forwards the first data to the front end page.

In the above embodiment, a data access request sent by an agent is received; the data access request is received by the agent terminal from a front-end page and is sent after the agent terminal is determined not to store first data corresponding to the data access request; calling an object storage end according to the data access request; and under the condition that the object storage end is determined to store the first data, feeding the first data back to the front-end page. In the embodiment of the application, the agent terminal and the object storage terminal are adopted for data storage, and when a data access request is sent by a front-end page, first data corresponding to the data access request can be searched for by the agent terminal; under the condition that the agent end does not store the first data, searching the first data from the object storage end; and under the condition that the object storage end stores the first data, feeding the first data back to the front-end page through the back end. Therefore, under the scene of high concurrent access, a large number of data access requests can access the agent end and the object storage end in sequence, so that the condition that a large number of data access requests access the database at the same time and great access pressure is caused to the database is avoided, high concurrent access requirements can be met, and the problem that the database crashes due to overlarge access pressure is avoided.

In an embodiment, the step of feeding back the first data to the front page may include: and under the condition that the first data is in the first data format, sending the first data to the proxy end for storage, and feeding the first data back to the front-end page through the proxy end.

The first data format is a static page.

The back end receives first data sent by the object storage end, and sends the static page to the agent end for storage under the condition that the first data is the static page; the proxy side forwards the static page to the front-end page.

In the above embodiment, the first data format is a static page, and when the first data format is the first data format, the first data is sent to the proxy terminal for storage, and the first data is fed back to the front-end page through the proxy terminal, and the first data is stored through the proxy terminal, so that the first data can be directly obtained from the proxy terminal without obtaining from the object storage terminal when the first data is accessed next time.

In an embodiment, the step of feeding back the first data to the front page may further include: and feeding back the first data to the front end page under the condition that the first data is in the second data format and is not empty.

And the second data format is a JSON data format.

The rear end receives first data sent by the object storage end, determines that the first data is JSON data, and determines whether the JSON data is empty; and under the condition that the JSON data is empty, the back end directly returns the JSON data to the front-end page.

In the above embodiment, when the first data is in the second data format and is not empty, the first data is fed back to the front-end page, so that the front-end page can be displayed according to the JSON data.

In an embodiment, fig. 5 is a flowchart of a data access method, which is described by taking the method as an example for a backend application, and may include the following steps:

s401, under the condition that the first data is in the second data format and is empty, third data are obtained from the database according to the data access request.

The rear end receives first data sent by the object storage end, determines that the first data is JSON data, and determines whether the JSON data is empty; and under the condition that the JSON data is not empty, the back end acquires the JSON data from the database according to the data access request.

S402, feeding back the third data to the front-end page through the proxy end, and sending the first data to the cache end for storage.

The back end returns the obtained JSON data to the agent end; and the proxy terminal forwards JSON data to a front-end page. The back end also asynchronously sends JSON data to the cache end; after receiving the JSON data, kafka producers in the cache end can monitor the JSON data in real time, and the Kafka consumers consume the JSON data monitored by the Kafka producers in real time and store the JSON data.

In the above embodiment, when the first data is in the second data format and is empty, the third data is obtained from the database according to the data access request; the third data are fed back to the front-end page through the proxy end, and the first data are sent to the cache end for storage, so that the third data can be directly obtained from the cache end when being accessed next time, the second data do not need to be obtained from the database again, and the pressure of the database is reduced.

On the basis of the above embodiments, the embodiments of the present application may further include the following steps: and receiving a data change request sent by the front-end page, and respectively sending fourth data corresponding to the data change request to the database and the cache end for storage.

The data change request refers to a change request of a detailed page, and due to the fact that the access quantity of the detailed page is not high and the change is not frequent, the data change request does not need to be updated to a static page every five minutes and only needs to be changed when the detailed page changes. The fourth data refers to data corresponding to detailed page changes.

And the back end receives a detailed page change request sent by the front end page, and sends data corresponding to the detailed page change request to the database to modify the database. In addition, the back end also asynchronously sends data corresponding to the detailed page change request to the cache end; and after receiving the data corresponding to the detailed page change request, the Kafka producer in the cache end can monitor in real time, and the Kafka consumer consumes the data corresponding to the detailed page change request monitored in real time by the Kafka producer, and further sends the data corresponding to the detailed page change request to the object storage end for storage.

In the above embodiment, the data change request sent by the front-end page is received, and the fourth data corresponding to the data change request is sent to the database and the cache terminal respectively for storage, so that not only is the database modified, but also the fourth data can be ensured to be directly obtained from the cache terminal when being accessed next time, and the pressure of the database is reduced.

In one embodiment, as shown in fig. 6, a process for data access is provided and applied to a data storage system, where the data storage system includes a proxy side, an object storage side, a cache side, a database, and a back end. The process comprises the following steps:

s501, the agent end receives the data access request sent by the front end page, and sends the data access request to the back end under the condition that the first data corresponding to the data access request is not stored.

The front-end page sends a data access request; and the proxy end receives the data access request, and sends the data access request to the backend under the condition that the first data corresponding to the data access request is determined not to be stored.

S502, the back end receives the data access request and calls the object storage end.

And the back end calls the object storage end according to the received data access request.

S503, the object storage end sends the first data to the back end.

And the object storage end returns the first data according to the data access request and sends the first data to the back end.

S504, under the condition that the first data is in the first data format, the back end sends the first data to the agent end.

The back end receives first data sent by the object storage end, and sends the static page to the proxy end under the condition that the first data is the static page.

And S505, the agent terminal stores the first data and feeds the first data back to the front-end page.

And the agent terminal stores the static page and forwards the static page to the front-end page.

S506, the back end feeds back the first data to the front end page when the first data is in the second data format and is not empty.

The rear end receives first data sent by the object storage end, determines that the first data is JSON data, and determines whether the JSON data is empty; and under the condition that the JSON data is not empty, the back end directly feeds the JSON data back to the front end page.

And S507, under the condition that the first data is in the second data format and is empty, the back end acquires third data from the database according to the data access request.

The rear end receives first data sent by the object storage end, determines that the first data is JSON data, and determines whether the JSON data is empty; and under the condition that the JSON data is empty, the back end acquires the JSON data from the database according to the data access request.

And S508, the back end feeds the third data back to the front end page through the proxy end, and sends the third data to the object storage end for storage.

The proxy end feeds the JSON data back to the front-end page, and forwards the JSON data to the front-end page; the proxy end asynchronously sends the JSON data to the cache end, and the cache end sends the JSON data to the object storage end for storage.

In the above embodiment, the proxy receives the data access request sent by the front-end page, and sends the data access request to the back-end under the condition that it is determined that the first data corresponding to the data access request is not stored; the back end receives the data access request and calls an object storage end; the object storage end sends the first data to the back end; under the condition that the first data is in a first data format, the back end sends the first data to the agent end; the back end stores the first data and feeds the first data back to the front end page; under the condition that the first data are in the second data format and are not empty, the back end feeds the first data back to the front end page; under the condition that the first data are in the second data format and are empty, the back end acquires third data from the database according to the data access request, and the third data pass through the proxy end; and the proxy end feeds the third data back to the front-end page and sends the third data to the cache end for storage. According to the embodiment of the application, IO interaction among systems is reduced through three-layer caching technology of the agent end, the object storage end and the cache end, and the access pressure of a database is relieved, so that high concurrent access requirements can be met.

It should be understood that, although the steps in the flowcharts related to the embodiments as described above are sequentially displayed as indicated by arrows, the steps are not necessarily performed sequentially as indicated by the arrows. The steps are not limited to being performed in the exact order illustrated and, unless explicitly stated herein, may be performed in other orders. Moreover, at least a part of the steps in the flowcharts related to the embodiments described above may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the execution order of the steps or stages is not necessarily sequential, but may be rotated or alternated with other steps or at least a part of the steps or stages in other steps.

Based on the same inventive concept, the embodiment of the present application further provides a data access device for implementing the above-mentioned data access method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the method, so specific limitations in one or more embodiments of the data access device provided below may refer to the limitations on the data access method in the foregoing, and details are not described here.

In one embodiment, as shown in fig. 7, there is provided a data access apparatus applied to an object storage side, including:

a calling request receiving module 601, configured to receive a calling request sent by a back end according to a data access request; the data access request is received by the agent terminal from a front-end page, and the agent terminal is sent to the back-end after determining that the agent terminal does not store first data corresponding to the data access request;

the data feedback module 602 is configured to, when it is determined that the first data is stored according to the call request, feed back the first data to the back end, so as to feed back the first data to the front-end page through the back end.

In one embodiment, the data feedback module 602 is specifically configured to, when the first data is in the first data format, send the first data to the agent terminal through the backend for storage, and feed the first data back to the front-end page through the agent terminal.

In one embodiment, the data feedback module 602 is specifically configured to receive second data sent by the front-end page according to a preset period, and store the second data.

In one embodiment, the data feedback module 602 is specifically configured to feed back the first data to the front-end page through the back end when the first data is in the second data format and is not empty.

In one embodiment, as shown in fig. 8, the apparatus further comprises:

the data storage module 603 is configured to store third data sent by the cache; and the third data is obtained from the database and sent to the cache terminal for storage according to the data access request under the condition that the back end determines that the first data is in the second data format and is empty.

In one embodiment, the data storage module 603 is further configured to store fourth data sent by the cache; the fourth data is sent to the cache end for storage after the back end receives the data change request sent by the front end page.

In one embodiment, as shown in fig. 9, the apparatus further comprises:

a discarding module 604, configured to discard the data access request if the data access request does not match a preset database access condition.

In one embodiment, as shown in fig. 10, the apparatus further comprises:

a first data obtaining module 605, configured to, if multiple requests for accessing the same data are received, obtain data according to a first request with the earliest access time, and send the obtained data to a cache for storage;

the second data obtaining module 606 is configured to obtain, from the cache terminal, data corresponding to a second request, except the first request, in the multiple data access requests.

In one embodiment, as shown in fig. 11, the apparatus further comprises:

the updating module 607 is configured to query the database for data and update the object storage end if the data in the object storage end is expired.

In one embodiment, as shown in fig. 12, there is provided a data access apparatus, applied to a backend, including:

a request receiving module 701, configured to receive a data access request sent by an agent; the data access request is received by the agent terminal from a front-end page and is sent after the agent terminal is determined not to store first data corresponding to the data access request;

a calling module 702, configured to call an object storage end according to the data access request;

the data feedback module 703 is configured to feed back the first data to the front-end page when it is determined that the object storage end stores the first data.

In one embodiment, the data feedback module 703 is specifically configured to send the first data to the agent for storage when the first data is in the first data format, and feed the first data back to the front-end page through the agent.

In one embodiment, the data feedback module 703 is specifically configured to feed back the first data to the front page when the first data is in the second data format and is not empty.

In one embodiment, as shown in fig. 13, the apparatus further comprises:

a data obtaining module 704, configured to obtain third data from the database according to the data access request when the first data is in the second data format and is empty;

the data storage module 705 is configured to feed back the third data to the front-end page through the proxy side, and send the first data to the cache side for storage.

In one embodiment, the data storage module 705 is further configured to receive a data change request sent by a front-end page, and send fourth data corresponding to the data change request to the database and the cache for storage, respectively.

The various modules in the data access device described above may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent of a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 14. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used to store data access data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data access method.

Those skilled in the art will appreciate that the architecture shown in fig. 14 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, comprising a memory having a computer program stored therein and a processor that when executing the computer program performs the steps of:

In one embodiment, the processor, when executing the computer program, further performs the steps of:

In one embodiment, the processor when executing the computer program further performs the steps of:

storing fourth data sent by a cache end; and the fourth data is sent to the cache end for storage after the back end receives the data change request sent by the front end page.

calling an object storage end according to the data access request;

and feeding back the first data to the front-end page under the condition that the object storage end is determined to store the first data.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high-density embedded nonvolatile Memory, resistive Random Access Memory (ReRAM), magnetic Random Access Memory (MRAM), ferroelectric Random Access Memory (FRAM), phase Change Memory (PCM), graphene Memory, and the like. Volatile Memory can include Random Access Memory (RAM), external cache Memory, and the like. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others. The databases involved in the embodiments provided herein may include at least one of relational and non-relational databases. The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing based data processing logic devices, etc., without limitation.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A data access method, applied to an object store, the method comprising:

receiving a calling request sent by a back end according to a data access request; the data access request is received by a proxy end from a front-end page, and the data access request is sent to the back end after the proxy end is determined not to store first data corresponding to the data access request;

2. The method of claim 1, wherein feeding back the first data to the front-end page by the back-end comprises:

and under the condition that the first data is in a first data format, sending the first data to the agent end for storage through the back end, and feeding back the first data to the front end page through the agent end.

3. The method of claim 2, further comprising:

4. The method of claim 1, wherein the feeding back, by the back end, the first data to a front end page comprises:

and under the condition that the first data is in a second data format and is not empty, feeding back the first data to the front-end page through the back end.

5. The method of claim 1, further comprising:

storing the third data sent by the cache end; and the third data is obtained from a database according to the data access request and sent to the cache terminal for storage when the back terminal determines that the first data is in the second data format and is empty.

6. The method according to claim 4 or 5, further comprising:

7. The method of claim 1, further comprising:

and if the data access request is not matched with the preset database access condition, discarding the data access request.

8. The method of claim 1, further comprising:

and acquiring data corresponding to a second request except the first request in the multiple data access requests from the cache end.

9. The method of claim 1, further comprising:

and if the data in the object storage end is out of date, inquiring the data from the database and updating the object storage end.

10. A data access method, applied to a backend, the method comprising:

calling an object storage end according to the data access request;

11. The method of claim 10, wherein feeding the first data back to a front end page comprises:

and under the condition that the first data is in a first data format, sending the first data to the proxy end for storage, and feeding back the first data to the front-end page through the proxy end.

12. The method of claim 10, wherein feeding back the first data to a front-end page comprises:

and feeding back the first data to the front end page under the condition that the first data is in a second data format and is not empty.

13. The method of claim 10, further comprising:

under the condition that the first data is in a second data format and is empty, acquiring third data from a database according to the data access request;

and feeding back the third data to the front-end page through the proxy end, and sending the first data to a cache end for storage.

14. The method of claim 10, further comprising:

and receiving a data change request sent by the front-end page, and respectively sending fourth data corresponding to the data change request to a database and a cache end for storage.

15. A data access apparatus, applied to an object store, the apparatus comprising:

and the first data feedback module is used for feeding back the first data to the back end under the condition that the first data are determined to be stored according to the calling request, so that the first data are fed back to the front-end page through the back end.

16. A data access apparatus, for use in a backend, the apparatus comprising:

17. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 14.

18. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 14.

19. A computer program product comprising a computer program, characterized in that the computer program realizes the steps of the method of any one of claims 1 to 14 when executed by a processor.