CN106934044B - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN106934044B CN106934044B CN201710156581.1A CN201710156581A CN106934044B CN 106934044 B CN106934044 B CN 106934044B CN 201710156581 A CN201710156581 A CN 201710156581A CN 106934044 B CN106934044 B CN 106934044B
- Authority
- CN
- China
- Prior art keywords
- data
- change operation
- level cache
- database
- cache
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/235—Update request formulation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a data processing method and a device, wherein the method comprises the following steps: when data in a database is subjected to change operation, corresponding change operation is executed on the data in the first-level cache and the data in the second-level cache according to the change operation; and executing corresponding change operation on the data in the database according to the operation file generated by the first-level cache change operation. By the technical scheme of the invention, the response speed of the system can be improved.
Description
Technical Field
The embodiment of the invention relates to the field of computers, in particular to a data processing method and device.
Background
With the increasing data volume, the data operation based on the traditional database can not meet the performance requirement, people pursue the system performance more and more, the stored data of the traditional database (mysql, oracle and the like) is safe, and the distributed memory storage such as redis can read and write data quickly. However, it is generally considered unsafe to store data as a persistent layer, and once the redis service goes down, the whole system will be down, while the conventional database service is more stable. Therefore, the two are combined into a whole, so that the processing capacity of mass data can be improved, the performance of a system can be improved, and the safety of the data can be ensured.
The existing solution is as follows: and a mysql database and redis used as a cache mechanism to cache the data which need to be shared in the distributed system, such as the verification code and the session information, and the data is time-efficient. The processing of the data operates directly on the database. Once the amount of data reaches a certain order of magnitude, the query will necessarily compromise the performance of the system as a whole.
The existing solution is as follows: the method comprises the steps that a redis is used as a first-level cache, all operations on data operate the redis cache firstly, then a database is operated, the data in the redis are synchronous with the data in the database, the redis is inquired firstly during inquiry to ensure the consistency of the data, if the data in the redis not existed, the database is inquired, the performance of a system can be partially improved, if the data in the database is consistent with the data in the redis, the inquiry is undoubtedly doubled to improve the performance of the system, but the database can be directly operated during the increasing and deleting modification, and the performance of increasing, deleting and modifying is still not ideal.
Disclosure of Invention
The invention provides a data processing method and a data processing device, which are used for improving the response speed of a system.
In a first aspect, an embodiment of the present invention provides a data processing method, including:
when data in a database is subjected to change operation, corresponding change operation is executed on the data in the first-level cache and the data in the second-level cache according to the change operation;
and executing corresponding change operation on the data in the database according to the operation file generated by the first-level cache change operation.
Further, when performing a change operation on data in the database, performing a corresponding change operation on the data in the first-level cache and the data in the second-level cache according to the change operation includes:
when data in a database is subjected to change operation, deleting the data in the secondary cache, wherein the change operation is at least one of addition, deletion and modification;
and executing corresponding change operation on the data in the first-level cache according to the change operation.
Further, the method also comprises the following steps:
when data is read from a database, whether corresponding data exists in the second-level cache is firstly inquired, and if the corresponding data exists, the data is directly read from the second-level cache.
Further, the method also comprises the following steps:
when data is read from a database, whether corresponding data exists in the second-level cache is firstly inquired, and if the corresponding data does not exist, the corresponding data is read from the first-level cache.
Further, the executing the corresponding change operation on the data in the database according to the operation file generated by the first-level cache change operation includes:
generating an operation file according to the change operation of the secondary cache;
analyzing the change operation of the primary cache according to the operation file;
and changing the data in the database according to the analyzed change operation of the primary cache.
Further, the second-level cache is ehcache, and the first-level cache is redis.
In a second aspect, an embodiment of the present invention further provides a data processing apparatus, where the apparatus includes:
the first execution module is used for executing corresponding change operation on the data in the first-level cache and the data in the second-level cache according to the change operation when the change operation is carried out on the data in the database;
and the second execution module is used for executing corresponding change operation on the data in the database according to the operation file generated by the first-level cache change operation.
Further, the first execution module is specifically configured to:
when data in a database is subjected to change operation, deleting the data in the secondary cache, wherein the change operation is at least one of addition, deletion and modification;
and executing corresponding change operation on the data in the first-level cache according to the change operation.
Further, the method also comprises the following steps:
the first query module is used for querying whether corresponding data exist in the second-level cache or not when data are read from a database, and directly reading the data from the second-level cache if the corresponding data exist.
Further, the method also comprises the following steps:
and the second query module is used for querying whether corresponding data exists in the second-level cache or not when data is read from the database, and reading the corresponding data from the first-level cache if the corresponding data does not exist.
Further, the second execution module is specifically configured to:
generating an operation file according to the change operation of the secondary cache;
analyzing the change operation of the primary cache according to the operation file;
and changing the data in the database according to the analyzed change operation of the primary cache.
Further, the second-level cache is ehcache, and the first-level cache is redis.
When data in a database is subjected to change operation, corresponding change operation is executed on the data in a first-level cache and the data in a second-level cache according to the change operation; and executing corresponding change operation on the data in the database according to the operation file generated by the first-level cache change operation, solving the problem of reducing the response speed caused by directly operating the database in the prior art, and achieving the effect of improving the response speed.
Drawings
Fig. 1 is a flowchart of a data processing method according to a first embodiment of the present invention;
FIG. 2 is a flow chart of a data processing method according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a data processing apparatus according to a third embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention, where the present embodiment is applicable to a data processing situation, and the method may be executed by a data processing apparatus according to an embodiment of the present invention, where the apparatus may be implemented in a software and/or hardware manner, as shown in fig. 1, the method specifically includes the following steps:
s110, when the data in the database is changed, corresponding changing operations are executed on the data in the first-level cache and the data in the second-level cache according to the changing operations.
Specifically, when data in the database is changed, the data in the second-level cache is deleted according to the change operation, and after the data in the second-level cache is deleted, the corresponding change operation is executed on the data in the first-level cache. For example, when the data in the database is deleted, the data in the second level cache is deleted, and after the data in the second level cache is deleted, the deletion operation is performed on the data in the first level cache. For example, when the data in the database is subjected to the increment operation, the data in the second-level cache is deleted, and after the data in the second-level cache is deleted, the increment operation is performed on the data in the first-level cache.
Optionally, when performing a change operation on data in the database, performing a corresponding change operation on the data in the first-level cache and the data in the second-level cache according to the change operation includes:
when data in a database is subjected to change operation, deleting the data in the secondary cache, wherein the change operation is at least one of addition, deletion and modification;
and executing corresponding change operation on the data in the first-level cache according to the change operation.
Optionally, the second-level cache is ehcache, and the first-level cache is redis.
Wherein the ehcache is a widely used open source Java distributed cache. Mainly oriented to general caches, Java EE and lightweight containers. The cache has the characteristics of memory and disk storage, a cache loader, cache expansion, a cache exception handling program, a gzip cache servlet filter, support of REST and SOAP api and the like. The redis is a key-value storage system, and the redis supports various different modes of sorting. To ensure efficiency, data is cached in memory. The redis periodically writes updated data into a disk or writes modification operation into an additional recording file, and realizes master-slave synchronization on the basis of the updated data or the modification operation. The occurrence of redis greatly compensates the deficiency of keyvalue storage such as memcached, and can play a good role in supplementing the relational database in some occasions. The PHP client is provided with Python, Ruby, Erlang and PHP clients, and is convenient to use.
And S120, executing corresponding change operation on the data in the database according to the operation file generated by the first-level cache change operation.
Optionally, the executing, according to the operation file generated by the first-level cache change operation, a corresponding change operation on data in the database includes: generating an operation file according to the change operation of the secondary cache;
analyzing the change operation of the primary cache according to the operation file;
and changing the data in the database according to the analyzed change operation of the primary cache.
In a specific example, each server a, server B, server C and server D uses a redis distributed cache as a primary cache, and the ehcache is a secondary cache, when data is subjected to an add operation, the server is randomly selected to perform an operation of deleting data in the ehcache, that is, a value corresponding to a key in the ehcache is deleted, so as to select data in the ehcache in the server a to be deleted first, the ehcache sends a message to the server B, the server C and the server D, the server B, the server C and the ehcache in the server D are notified to perform a data deletion operation, the server a modifies data in the redis, other servers are notified to modify data in the redis through a distributed management framework, when the redis is operated, the redis generates an operation file, the operation file is detected in real time, and the operation of the data in the redis is determined as an add operation according to the operation file, Deleting operation or changing operation, further changing the data in the database according to the operation condition of the data in the redis, and if the operation of the data in the redis is determined to be adding operation through the operation file, adding operation is carried out on the data in the database; and if the operation of the data in the redis is determined to be deleting operation through the operation file, deleting operation is carried out on the data in the database.
According to the technical scheme of the embodiment, when data in the database is subjected to change operation, corresponding change operation is performed on the data in the first-level cache and the data in the second-level cache according to the change operation; and executing corresponding change operation on the data in the database according to the operation file generated by the first-level cache change operation, solving the problem of reducing the response speed caused by directly operating the database in the prior art, and achieving the effect of improving the response speed.
Example two
Fig. 2 is a flowchart of a data processing method according to a second embodiment of the present invention, which is optimized based on the second embodiment, and in this embodiment, the method further includes: when data is read from a database, whether corresponding data exists in the second-level cache is firstly inquired, and if the corresponding data exists, the data is directly read from the second-level cache.
Therefore, when data is read from a database, whether corresponding data exists in the second-level cache is inquired firstly, if the corresponding data exists, the data is directly read from the second-level cache, the situation that the response speed is reduced due to the fact that the database is inquired directly when the data is read from the database is avoided, the second-level cache is inquired firstly due to the fact that the data reading and writing speed of the second-level cache is very high, and if the data exists in the second-level cache, the data in the second-level cache is read directly, and the response speed is improved.
As shown in fig. 2, the method provided in the embodiment of the present invention specifically includes the following steps:
s210, when the data in the database is changed, corresponding changing operations are executed on the data in the first-level cache and the data in the second-level cache according to the changing operations.
S220, executing corresponding change operation on the data in the database according to the operation file generated by the first-level cache change operation.
And S230, when data is read from the database, firstly inquiring whether corresponding data exists in the second-level cache, and if the corresponding data exists, directly reading the data from the second-level cache.
Optionally, the method further includes:
when data is read from a database, whether corresponding data exists in the second-level cache is firstly inquired, and if the corresponding data does not exist, the corresponding data is read from the first-level cache.
In a specific example, under the condition that data needs to be queried, whether the data in the local ehcache contains the data needing to be queried is queried first, if the data contains the data needing to be queried, the data in the ehcache is called directly for reading, and if the data needing to be queried is not queried in the local ehcache, whether the data in the redis contains the data needing to be queried is queried, and the data in the redis is called for reading.
According to the technical scheme, whether corresponding data exist in the second-level cache or not is firstly inquired when data are read from a database, if the corresponding data exist, the data are directly read from the second-level cache, the problem that the response speed is reduced due to the fact that the database is directly inquired when the data are read from the database is solved, the second-level cache is firstly inquired due to the fact that the data reading and writing speed of the second-level cache is very high, and if the data exist in the second-level cache, the data in the second-level cache are directly read, and the technical effect of improving the response speed is achieved.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a data processing apparatus according to a third embodiment of the present invention. The present embodiment may be applicable to the case of data processing, the system may be implemented in a software and/or hardware manner, the system may be integrated in any device providing data processing, as shown in fig. 3, and the data processing apparatus specifically includes: a first execution module 310 and a second execution module 320.
The first executing module 310 is configured to, when a change operation is performed on data in the database, execute a corresponding change operation on the data in the first-level cache and the data in the second-level cache according to the change operation;
the second executing module 320 is configured to execute a corresponding change operation on data in the database according to the operation file generated by the first-level cache change operation.
Optionally, the first executing module 310 is specifically configured to:
when data in a database is subjected to change operation, deleting the data in the secondary cache, wherein the change operation is at least one of addition, deletion and modification;
and executing corresponding change operation on the data in the first-level cache according to the change operation.
Optionally, the method further includes:
the first query module is used for querying whether corresponding data exist in the second-level cache or not when data are read from a database, and directly reading the data from the second-level cache if the corresponding data exist.
Optionally, the method further includes:
and the second query module is used for querying whether corresponding data exists in the second-level cache or not when data is read from the database, and reading the corresponding data from the first-level cache if the corresponding data does not exist.
Optionally, the second executing module 320 is specifically configured to:
generating an operation file according to the change operation of the secondary cache;
analyzing the change operation of the primary cache according to the operation file;
and changing the data in the database according to the analyzed change operation of the primary cache.
Optionally, the second-level cache is ehcache, and the first-level cache is redis.
According to the technical scheme of the embodiment, when data in the database is subjected to change operation, corresponding change operation is performed on the data in the first-level cache and the data in the second-level cache according to the change operation; and executing corresponding change operation on the data in the database according to the operation file generated by the first-level cache change operation, solving the problem of reducing the response speed caused by directly operating the database in the prior art, and achieving the effect of improving the response speed.
The product can execute the method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (8)
1. A data processing method, comprising:
when data in a database is subjected to change operation, corresponding change operation is executed on the data in the first-level cache and the data in the second-level cache according to the change operation;
executing corresponding change operation on the data in the database according to the operation file generated by the first-level cache change operation;
when data in a database is subjected to change operation, corresponding change operation is executed on the data in the first-level cache and the data in the second-level cache according to the change operation, and the method comprises the following steps:
when data in a database is subjected to change operation, deleting the data in the secondary cache, wherein the change operation is at least one of addition, deletion and modification;
executing corresponding change operation on the data in the first-level cache according to the change operation;
executing corresponding change operation on the data in the database according to the operation file generated by the first-level cache change operation, wherein the corresponding change operation comprises the following steps:
generating an operation file according to the change operation of the first-level cache;
analyzing the change operation of the primary cache according to the operation file;
and changing the data in the database according to the analyzed change operation of the primary cache.
2. The method of claim 1, further comprising:
when data is read from a database, whether corresponding data exists in the second-level cache is firstly inquired, and if the corresponding data exists, the data is directly read from the second-level cache.
3. The method of claim 1, further comprising:
when data is read from a database, whether corresponding data exists in the second-level cache is firstly inquired, and if the corresponding data does not exist, the corresponding data is read from the first-level cache.
4. The method of claim 1, wherein the second level cache is ehcache and the first level cache is redis.
5. A data processing apparatus, comprising:
the first execution module is used for executing corresponding change operation on the data in the first-level cache and the data in the second-level cache according to the change operation when the change operation is carried out on the data in the database;
the second execution module is used for executing corresponding change operation on the data in the database according to the operation file generated by the first-level cache change operation;
the first execution module is specifically configured to:
when data in a database is subjected to change operation, deleting the data in the secondary cache, wherein the change operation is at least one of addition, deletion and modification;
executing corresponding change operation on the data in the first-level cache according to the change operation;
the second execution module is specifically configured to:
generating an operation file according to the change operation of the first-level cache;
analyzing the change operation of the primary cache according to the operation file;
and changing the data in the database according to the analyzed change operation of the primary cache.
6. The apparatus of claim 5, further comprising:
the first query module is used for querying whether corresponding data exist in the second-level cache or not when data are read from a database, and directly reading the data from the second-level cache if the corresponding data exist.
7. The apparatus of claim 5, further comprising:
and the second query module is used for querying whether corresponding data exists in the second-level cache or not when data is read from the database, and reading the corresponding data from the first-level cache if the corresponding data does not exist.
8. The apparatus of claim 5, wherein the second level cache is ehcache and the first level cache is redis.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710156581.1A CN106934044B (en) | 2017-03-16 | 2017-03-16 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710156581.1A CN106934044B (en) | 2017-03-16 | 2017-03-16 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106934044A CN106934044A (en) | 2017-07-07 |
| CN106934044B true CN106934044B (en) | 2020-02-14 |
Family
ID=59432578
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710156581.1A Active CN106934044B (en) | 2017-03-16 | 2017-03-16 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106934044B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108833495A (en) * | 2018-05-24 | 2018-11-16 | 努比亚技术有限公司 | Data cache method, equipment and the computer readable storage medium of user's request |
| CN110244912B (en) * | 2019-06-20 | 2022-06-21 | 上海数据交易中心有限公司 | Caching method of data distribution system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101046807A (en) * | 2006-03-31 | 2007-10-03 | 华为技术有限公司 | Method and device of storage data readed |
| CN103116627A (en) * | 2013-01-31 | 2013-05-22 | 汉柏科技有限公司 | Database access method with high concurrency service-oriented architecture (SOA) technology and system |
| CN104021192A (en) * | 2014-06-13 | 2014-09-03 | 北京联时空网络通信设备有限公司 | Database renewing method and device |
| CN104424275A (en) * | 2013-08-29 | 2015-03-18 | 中兴通讯股份有限公司 | Database system and data synchronization method |
| CN105183394A (en) * | 2015-09-21 | 2015-12-23 | 北京奇虎科技有限公司 | Data storage processing method and device |
| CN106446037A (en) * | 2016-08-31 | 2017-02-22 | 南威软件股份有限公司 | Method for realizing consistency of Redis and MYSQL data based on distributed lock |
-
2017
- 2017-03-16 CN CN201710156581.1A patent/CN106934044B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101046807A (en) * | 2006-03-31 | 2007-10-03 | 华为技术有限公司 | Method and device of storage data readed |
| CN103116627A (en) * | 2013-01-31 | 2013-05-22 | 汉柏科技有限公司 | Database access method with high concurrency service-oriented architecture (SOA) technology and system |
| CN104424275A (en) * | 2013-08-29 | 2015-03-18 | 中兴通讯股份有限公司 | Database system and data synchronization method |
| CN104021192A (en) * | 2014-06-13 | 2014-09-03 | 北京联时空网络通信设备有限公司 | Database renewing method and device |
| CN105183394A (en) * | 2015-09-21 | 2015-12-23 | 北京奇虎科技有限公司 | Data storage processing method and device |
| CN106446037A (en) * | 2016-08-31 | 2017-02-22 | 南威软件股份有限公司 | Method for realizing consistency of Redis and MYSQL data based on distributed lock |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106934044A (en) | 2017-07-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6535031B2 (en) | Data query method and apparatus | |
| US11709803B2 (en) | Data transaction processing method, apparatus, and electronic device | |
| US11836112B2 (en) | Path resolver for client access to distributed file systems | |
| US10970311B2 (en) | Scalable snapshot isolation on non-transactional NoSQL | |
| CN110737682A (en) | cache operation method, device, storage medium and electronic equipment | |
| US11822912B2 (en) | Software installation through an overlay file system | |
| US11216421B2 (en) | Extensible streams for operations on external systems | |
| US11113249B2 (en) | Multitenant application server using a union file system | |
| US11620310B1 (en) | Cross-organization and cross-cloud automated data pipelines | |
| US20190057124A1 (en) | Key Ticketing System with Lock-Free Concurrency and Versioning | |
| CN109992469B (en) | Method and device for merging logs | |
| CN106934044B (en) | Data processing method and device | |
| US11429311B1 (en) | Method and system for managing requests in a distributed system | |
| CN111475279B (en) | System and method for intelligent data load balancing for backup | |
| US10073657B2 (en) | Data processing apparatus, data processing method, and computer program product, and entry processing apparatus | |
| JP2017515236A (en) | Method and apparatus for hard disk to execute application code | |
| CN107590199B (en) | Memory-oriented multithreading database design method | |
| US10067678B1 (en) | Probabilistic eviction of partial aggregation results from constrained results storage | |
| US10762139B1 (en) | Method and system for managing a document search index | |
| US20220382712A1 (en) | Minimizing data volume growth under encryption changes | |
| US20130282654A1 (en) | Query engine communication | |
| JP2011186853A (en) | Data processing device, system, method and program | |
| US11113296B1 (en) | Metadata management for a transactional storage system | |
| US11816088B2 (en) | Method and system for managing cross data source data access requests | |
| US20240195877A1 (en) | Session based data persistency |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 100193 5th floor 510, No. 5 Building, East Yard, No. 10 Wangdong Road, Northwest Haidian District, Beijing Patentee after: Beijing Shendun Technology Co.,Ltd. Address before: 100193 5th floor 510, No. 5 Building, East Yard, No. 10 Wangdong Road, Northwest Haidian District, Beijing Patentee before: BEIJING SENSESHIELD TECHNOLOGY Co.,Ltd. |
|
| CP01 | Change in the name or title of a patent holder |