CN110427364A

CN110427364A - A kind of data processing method, device, electronic equipment and storage medium

Info

Publication number: CN110427364A
Application number: CN201910544603.0A
Authority: CN
Inventors: 类兴邦
Original assignee: Beijing QIYI Century Science and Technology Co Ltd
Current assignee: Beijing QIYI Century Science and Technology Co Ltd
Priority date: 2019-06-21
Filing date: 2019-06-21
Publication date: 2019-11-08

Abstract

The present invention provides a kind of data processing method, device, electronic equipment and storage mediums.The described method includes: obtaining the corresponding data to be replaced of the target data when monitoring modification corresponding with target data operation；According to the data routing table being pre-created, searches and indexed with the matched target of the target data；The mapping relations between write-in data and index are preserved in the data routing table；It is indexed according to the target, the data to be replaced is routed into the target and index corresponding target storage space；In the target storage space, the target data is replaced with into the data to be replaced.The present invention can need to have been written into modify when, directly according to the mapping relations between write-in data and index, determine that the routing of replacement data is directed toward, and corresponding data be replaced, it is thus possible to reduce redundant data to the occupancy of memory source.

Description

A kind of data processing method, device, electronic equipment and storage medium

Technical field

The present invention relates to technical field of data processing, more particularly to a kind of data processing method, device, electronic equipment and Storage medium.

Background technique

Elasticsearch (abbreviation ES) is the search server for being based on Lucene (i.e. full-text search engine), it The full-text search engine of a distributed multi-user ability is provided, RESTful web, java interface is supported, can support reality When search for, have stablize, reliably, quickly, it is easy to install and use the features such as.

Fragment is the basic storage cell of each index data table of ES, and bottom is a Lucene storage catalogue, can be with It is distributed to different memory nodes.And when previously positioned fragment data amount is excessive, but legacy data due to business need need Asking does not allow to delete, or could delete after a period of time, so as to form mass data.And for Yi Dan be written it is subsequent once not The time series data (such as operation log) modified again, ES provide a kind of rollover mode, and threshold values, every rope can be set in user When drawing corresponding fragment and reaching threshold values, new index can be rolled out automatically, and newest index is directed toward in the reference of index.And this side Formula is only capable of rolling out new index, and after rolling out new index, the data of subsequent acquisition are the fragment storages being directed toward according to new index Data, even if being also the fragment being directed toward according to new index for obtained modification data there is a situation where modifying legacy data Data storage is carried out, thus, it cannot be modified for having been written into the data in fragment still, can only realize and modification data are deposited again Storage.

In above scheme, under the scene of mass data write-in, above scheme will be unable to meet the demand, subsequent dilatation, It can only be by way of rebuilding index, and huge memory source will certainly be expended by rebuilding index.

Summary of the invention

The embodiment of the invention provides a kind of data processing method, device, electronic equipment and storage mediums, existing to solve In technology when a fragment data amount is excessive, ES does not have the function of dividing automatically, leads to inquiry and the warehouse-in efficiency of system The technical issues of reduction.

To solve the above-mentioned problems, the embodiment of the invention provides a kind of data processing methods, comprising:

When monitoring modification corresponding with target data operation, the corresponding data to be replaced of the target data are obtained；

According to the data routing table being pre-created, searches and indexed with the matched target of the target data；The data road By preserving the mapping relations between write-in data and index in table；

It is indexed according to the target, the data to be replaced is routed into the target and index corresponding target storage sky Between；

In the target storage space, the target data is replaced with into the data to be replaced.

Preferably, before the corresponding data to be replaced of the acquisition target data, further includes:

Whether the first data volume in corresponding first memory space of the first index of detection reaches the first data-quantity threshold；

In the case where first data volume reaches first data-quantity threshold, creation is associated with first index The second index, and the second memory space corresponding with second index；The target index is first index and institute An index in the second index is stated, the target storage space is in first memory space and second memory space A memory space；

Obtain data to be written；

According to second index, second memory space is written into the data to be written.

Preferably, whether the first data volume in corresponding first memory space of first index of detection reaches the first number According to amount threshold value, comprising:

Every prefixed time interval, the first number in corresponding first memory space of primary the first index of the detection is executed The step of whether reaching the first data-quantity threshold according to amount.

Preferably, whether the first data volume in corresponding first memory space of first index of detection reaches first Before data-quantity threshold, further includes:

Create the data routing table；

Obtain first index, and the first data being stored in first memory space；

The first mapping relations between first index and first data are established, and by first mapping relations It saves to the data routing table；

It is also wrapped after second memory space is written in the data to be written described according to second index It includes:

The second mapping relations between the data to be written and second index are established, and second mapping is closed System saves to the data routing table.

Preferably, after the acquisition data to be written, further includes:

By the data buffer storage to be written in spatial cache, to stop that data are written into first memory space.

Preferably, second memory space includes the first memory and the second memory, described according to second index, will Second memory space is written in the data to be written, comprising:

According to second index, the data persistence to be written is stored in first memory, and will it is described to Data buffer storage is written in second memory.

To solve the above-mentioned problems, the embodiment of the invention provides a kind of data processing equipments, comprising:

Data acquisition module to be replaced, for obtaining the mesh when monitoring modification corresponding with target data operation Mark the corresponding data to be replaced of data；

Target indexes searching module, for according to the data routing table being pre-created, lookup to be matched with the target data Target index；The mapping relations between write-in data and index are preserved in the data routing table；

The data to be replaced are routed to the mesh for indexing according to the target by data routing module to be replaced Mark indexes corresponding target storage space；

Target data replacement module, in the target storage space, by the target data replace with it is described to Replacement data.

Preferably, described device further include:

First data volume detection module is for detecting the first data volume in the first corresponding first memory space of index It is no to reach the first data-quantity threshold；

Second index creation module, in the case where first data volume reaches first data-quantity threshold, Creation and associated second index of first index, and the second memory space corresponding with second index；The target Index is an index in first index and second index, and the target storage space is that first storage is empty Between and second memory space in a memory space；

Data acquisition module to be written, for obtaining data to be written；

Data write. module to be written, for according to second index, the data to be written to be written described second Memory space.

Preferably, the first data volume detection module includes:

Implementation sub-module is detected, for executing the primary first data volume detection module every prefixed time interval.

Preferably, described device further include:

Data routing table creation module, for creating the data routing table；

First data acquisition module, for obtaining first index, and the be stored in first memory space One data；

First mapping relations establish module, the first mapping for establishing between first index and first data Relationship, and first mapping relations are saved to the data routing table；

Described device further include:

Second mapping relations establish module, and second for establishing between the data to be written and second index is reflected Relationship is penetrated, and second mapping relations are saved to the data routing table.

Preferably, described device further include:

Data stop writing module, are used for the data buffer storage to be written in spatial cache, to stop to described first Data are written in memory space.

Preferably, second memory space includes the first memory and the second memory, the Data write. module to be written Include:

Data sub-module stored to be written, for according to second index, the data persistence to be written to be stored In first memory, and by the data buffer storage to be written in second memory.

To solve the above-mentioned problems, the embodiment of the invention provides a kind of electronic equipment, comprising: memory, processor and It is stored in the computer program that can be run on the memory and on the processor, the computer program is by the processing Device realizes data processing method described in any of the above embodiments when executing.

To solve the above-mentioned problems, the embodiment of the invention provides a kind of computer readable storage medium, the computers It is stored with computer program on readable storage medium storing program for executing, is realized described in any of the above-described when the computer program is executed by processor Data processing method.

Compared with prior art, the embodiment of the present invention includes following advantages:

In the embodiment of the present invention, by obtaining target data pair when monitoring modification corresponding with target data operation The data to be replaced answered are searched and are indexed with the matched target of target data according to the data routing table being pre-created, data routing The mapping relations between write-in data and index are preserved in table, and data to be replaced are routed into target index according to target index Corresponding target storage space, and in target storage space, target data is replaced with into data to be replaced.The embodiment of the present invention By creating data routing table, and the mapping relations saved in data routing table between write-in data and index in turn can be with Need to have been written into modify when, directly according to write-in data and index between mapping relations, determine replacement data Routing is directed toward, and is replaced to corresponding data, it is thus possible to reduce redundant data to the occupancy of memory source.

It should be understood that above general description and following detailed description be only it is exemplary and explanatory, not The application can be limited.

Detailed description of the invention

Fig. 1 is a kind of step flow chart of data processing method provided in an embodiment of the present invention；

Fig. 2 is a kind of step flow chart of data processing method provided in an embodiment of the present invention；

Fig. 3 is a kind of schematic diagram of data handling procedure provided in an embodiment of the present invention；

Fig. 4 is a kind of structural schematic diagram of data processing equipment provided in an embodiment of the present invention；

Fig. 5 is a kind of structural schematic diagram of data processing equipment provided in an embodiment of the present invention.

Specific embodiment

In order to make the foregoing objectives, features and advantages of the present invention clearer and more comprehensible, with reference to the accompanying drawing and specific real Applying mode, the present invention is described in further detail.

The term used in embodiments of the present invention is only to be not intended to be limiting merely for for the purpose of describing particular embodiments The present invention.In the embodiment of the present invention and the "an" of singular used in the attached claims, " described " and "the" It is also intended to including most forms, unless the context clearly indicates other meaning.

It will be appreciated that though index and data may be described using term first, second etc. in embodiments of the present invention Amount, but these indexes and data volume should not necessarily be limited by these terms.These terms are only used to index and data volume is distinguished from each other out. For example, the first data volume can also be referred to as the second data volume, similar in the case where not departing from range of embodiment of the invention Ground, the second data volume can also be referred to as the first data volume.

Referring to Fig.1, a kind of step flow chart of data processing method provided in an embodiment of the present invention is shown, it specifically can be with Include the following steps:

Step 101: when monitoring modification corresponding with target data operation, it is corresponding wait replace to obtain the target data Change data.

The embodiment of the present invention can be applied in the scene modified to the Elasticsearch data of write-in.

The related notion of ElasticSearch is described below below.

Elasticsearch is a kind of non-relational database, can solve some inquiry problems, Elasticsearch It can be used as an independent single search server, and in order to handle large data collection, realize fault-tolerant and high availability, ElasticSearch may operate on multiple servers cooperated with each other, and the collection of these servers is collectively referred to as cluster.

Node: each server for forming cluster is known as node.

Fragment: a fragment is the working cell of a bottom, is the container of data, and data can be stored in fragment. Elasticsearch can use fragment will be at node each in data distribution to cluster.It is more than single that index, which may store a large amount of, The data of the hardware limitation of node, for example, occupying the single index of 1,000,000,000 documents of 1TB disk space may be not suitable for individually The disk of node, in order to solve this problem, Elasticsearch, which is provided, is subdivided into multiple function for being known as fragment for index Energy.When creation index, required fragment number need to be only defined.Each fragment itself is one multiple functional and independent " index ", can be on any node of trustship in the cluster.

When creating index in ElasticSearch, one or more fragments can be distributed for the index of creation, one or Multiple fragments have collectively constituted memory space (target storage space referred in such as following step) corresponding with index, Ke Yili Xie Di, each fragment both correspond to a node in a cluster, i.e., configure a fragment on each node, and index can be with These fragments are directed toward, after obtaining data to be written, data can be written into, these fragments are written.

Index (index) refer to one of relational database individually, physics, to one or more columns per page in database table A kind of storage organization for being ranked up of value.In ElasticSearch, index is the set of one group of data, can by index Directly to find specified data.

Target data refers to the data that the needs having been written into memory space are modified.

Target data can be one or more daily record data, for example, for user's barrage msu message, certain bullet Curtain passes through in audit before, but changes auditing standards later, needs to be set as not passing through by barrage audit state, then this bar bullet Curtain is considered as target data.

Modification operation refers to the operation for modifying target data.Modification operation, which can be, triggers generation by business personnel Operation, such as above-mentioned example, business personnel can change barrage auditing standards, and then system can be audited according to the barrage after change Standard modifies to corresponding data, then the operation for receiving the barrage auditing standards of business personnel's change is considered as target data Modification operation.

Data to be replaced, which refer to, carries out target data after accordingly modifying, and the obtained number for being used to replace target data According to auditing the barrage passed through before such as above-mentioned example are as follows: " barrage content: aaabbb, barrage state: audit passes through ", by becoming It after more barrage auditing standards, needs the status modifier of the barrage to be that the audit fails, i.e. " barrage content: aaabbb, audit State: the audit fails ", then in above-mentioned example, " barrage content: aaabbb, barrage state: audit passes through " is number of targets According to " barrage content: aaabbb, audit state: the audit fails " is data to be replaced.

Certainly, in the present invention, when the number of target data is n, then the number and target data of data to be replaced Number it is identical, that is, be all n item, wherein n is positive integer more than or equal to 1.

Monitoring programme corresponding with operation is modified can be preset in systems, and business personnel's triggering can be monitored in real time Modification operation can be obtained from server and target data pair when monitoring modification corresponding with target data operation The initial data answered in turn modifies to initial data according to modification operation, to obtain the number to be replaced of replacement target data According to for example, the barrage of publication are as follows: " aaabbbccc ", the target barrage for having been written into memory space is that " aaabbbccc, audit are logical Cross ", when the status modifier of the barrage is is the state that the audit fails by modification operation, available original barrage, i.e., " aaabbbccc ", and modify according to modification operation to original barrage, obtain barrage to be replaced: " aaabbbccc, audit is not Pass through ".

It is to be appreciated that above-mentioned example is merely to more fully understand the technical solution of the embodiment of the present invention and showing for enumerating Example, not as the sole limitation to the embodiment of the present invention.

After obtaining data to be replaced corresponding with target data, step 102 is executed.

Step 102: according to the data routing table being pre-created, searching and indexed with the matched target of the target data；Institute State the mapping relations preserved between write-in data and index in data routing table.

Data routing table refers to the tables of data for saving the mapping relations between write-in data and index, for example, index Including index 1 and index 2, each data having been written into both correspond to only one Data Identification, for distinguishing the difference of write-in Data, Data Identification can be the mark i.e. specified when creating index, and Data Identification includes mark 1~1000 and mark 10001~2000, can saved in data routing table index 1 and mark 1~1000 between mapping relations, and index 2 with Mapping relations between mark 1001~2000.

Target index refers to the index with target data there are mapping relations, due to having pre-saved in data routing table The mapping relations between data and index are written, it, can be from the number being pre-created after determining the target data that needs are modified It is indexed according to being searched in routing table with the matched target of target data, such as above-mentioned example, is 2008 in the Data Identification of target data When, then it is index 2 with the matched index of Data Identification 2008, then indexing 2 is target index.

It is to be appreciated that above-mentioned example is merely to better understand the technical solution of the embodiment of the present invention and showing for enumerating Example, not as the sole limitation to the embodiment of the present invention.

It is finding with after the matched target index of target data, is executing step 103.

Step 103: being indexed according to the target, the data to be replaced are routed into the target and index corresponding target Memory space.

Target storage space refers to that target storage space can for storing the memory space for indexing matched data with target To be disk storage space, it is also possible to storage space of database, specifically, can be depending on business demand, the present invention is real It is without restriction to this to apply example.

Create target index when, can be pre-configured with it is corresponding with target index one or more fragment, one or more A fragment has collectively constituted target and has indexed corresponding memory space, i.e. target storage space.

After finding target index corresponding with target data in data routing table, can be indexed according to target, The routing for changing data to be replaced corresponding with target data is directed toward, i.e., the routing direction of data to be replaced is changed into target and deposited Space is stored up, i.e., is indexed according to target, data to be replaced are routed into target and index corresponding target storage space.

In existing program, after new index is rotated out in ElasticSearch, in subsequent generation data modification operation When, it is directed toward new memory space due to newly indexing, even if getting the data to be replaced of target data (data having been written into), It is to store data to be replaced to new memory space according to the direction newly indexed.

And the present invention is directed to establish data routing table, it is intended to obtain the corresponding rope of data (i.e. target data) for needing to modify Draw (i.e. target index), the to be replaced of the data for needing to modify can be changed by the target index obtained from data routing table The routings of data is directed toward, i.e., data to be replaced is routed to target and index corresponding memory space, so as to complete to having write Enter the modification of data, for example, current cable is cited as index 1, index 1 corresponds to memory space 1, that is, the data to be written obtained are by rope Draw in 1 direction memory space 1, and when modifying to certain data having been written into, it can be got according to data routing table The corresponding index of the data: index 2, index 2 correspond to memory space 2, corresponding wait replace in the data for obtaining with needing to modify It changes after data, is data to be replaced routing to be directed to by memory space 1 by index 1, and increasing data in existing program After routing table, the routing that by the index 2 obtained from data routing table, can change data to be replaced is directed toward, that is, is passed through Data to be replaced routing is directed to memory space 2 by index 2, so as to which data to be replaced are routed to memory space 2.

After data to be replaced to be routed to target according to target index and index corresponding target storage space, step is executed Rapid 104.

Step 104: in the target storage space, the target data being replaced with into the data to be replaced.

After data to be replaced are routed to target storage space, target data can be replaced with to data to be replaced, Specifically replacement process can be with are as follows: is the corresponding Data Identification of data grant target data to be replaced, and by target data from mesh It is removed in mark memory space.

The embodiment of the present invention is indexed by target and is changed by searching for the corresponding target index of target data for needing to modify The routings of data to be replaced is directed toward, and in turn, data to be replaced is routed to the corresponding target storage space of target data, complete to Replacement data replaces the operation of target data, realizes the modification to the legacy data having been written into, it is possible to reduce the redundancy in memory Data.

Data processing method provided in an embodiment of the present invention, by monitoring modification operation corresponding with target data When, the corresponding data to be replaced of target data are obtained, according to the data routing table being pre-created, are searched matched with target data Target indexes, and the mapping relations between write-in data and index are preserved in data routing table, will be to be replaced according to target index Data route to target and index corresponding target storage space, and in target storage space, target data is replaced with wait replace Change data.The embodiment of the present invention is being saved between write-in data and index in data routing table by creation data routing table Mapping relations, in turn, can need to have been written into modify when, directly according to write-in data and index between mapping Relationship determines that the routing of replacement data is directed toward, and is replaced to corresponding data, it is thus possible to which it is internal to reduce redundant data Deposit the occupancy of resource.

Referring to Fig. 2, a kind of step flow chart of data processing method provided in an embodiment of the present invention is shown, it specifically can be with Include the following steps:

Step 201: creating the data routing table.

Step 202: when monitoring modification corresponding with target data operation, it is corresponding wait replace to obtain the target data Change data.

The embodiment of above-mentioned steps 202 is similar to the embodiment of step 101 in above-described embodiment, the embodiment of the present invention It is not repeated here herein.

Step 202: obtaining first index, and the first data being stored in first memory space.

First index refers to a rope being just already present in Elasticsearch before creating data routing table Draw.

First memory space refers to for storing the memory space with the first matched data of index.First memory space can Think persistent storage space, such as mongo database, or spatial cache, such as CouchBase database.

First data refer to the data having been written into the first memory space.It is to be appreciated that the first data herein can be with It is a data, a plurality of data can also be made, it is generally the case that the data having been written into is a plurality of data, the embodiment of the present invention pair This is without restriction.

In Elasticsearch creation first index when, can also be the first index creation one or more fragment, one A or multiple fragments constitute corresponding first memory space of the first index, and after the index of creation first, the first index The data for needing to be written can be directed toward to the first memory space (i.e. one or more fragments).

It is to be appreciated that creation first index after, can be set the corresponding Data Identification of the first index since 0 to Infinity, and during constantly the first memory space is written in data, pieces of data according to Data Identification from 0,1 ... no It is disconnected to increase, and when the data volume of the first memory space reaches threshold value, then it is corresponding to record the data stored in the first memory space Data Identification.

Certainly, it when obtaining the first data, can be obtained according to the first index being pre-created from the first memory space The corresponding Data Identification of multiple first data.

It, can be according to corresponding first memory space of the first index, from the first memory space after the index of acquisition first The first data of middle acquisition, and execute step 203.

Step 203: establishing the first mapping relations between first index and first data, and by described first Mapping relations are saved to the data routing table.

First mapping relations refer to the mapping relations between the first index and the first data.

Index is usually to be identified by lowercase, to be distinguish to different indexes, such as indexes 1 corresponding rope Draw and be identified as a, the corresponding index of index 2 is identified as b etc..

The first data having been written into the first memory space can be by corresponding Data Identification, it is generally the case that for every Data is identified using ID, and different written data corresponds to different Data Identifications, for example, the first data include data A, the corresponding Data Identification of data B and data C, data A is 1, and the corresponding Data Identification of data B is 2, the corresponding data of data C It is identified as 3 etc..

It, can be according to the corresponding index mark of the first index and the first number after the index of acquisition first and the first data According to corresponding Data Identification, the first mapping relations between index mark and Data Identification are established, and by first mapping relations When being stored in data routing table, and modifying data in the follow-up process, the first data that can directly modify as needed are obtained First with mapping relations is taken to index, and the routing according to the first index change replacement data is directed toward, and is completed to needing to modify The first data modification operation.

Step 204: whether the first data volume in corresponding first memory space of the first index of detection reaches the first data Measure threshold value.

First data volume refers to the data volume of currently stored data in the first memory space.

First data-quantity threshold refers to preset data-quantity threshold corresponding with the first memory space, the first data volume The setting of threshold value can be set according to the size of the first memory space, for example, the size in the first memory space is 1000 When million, the first data-quantity threshold can be set as 800,000,000 or 900,000,000 etc., specifically, can be by for the first data-quantity threshold Business personnel sets as desired according to the size of the first memory space, and the embodiments of the present invention are not limited thereto.

In the present invention, the first data in corresponding first memory space of the first index can be detected every preset time Amount, and the first data volume that will test is compared with the first data-quantity threshold, to determine whether the first data volume reaches First data-quantity threshold, for example, the first data-quantity threshold is 1,000,000,000, the first data volume of detection is 800,000,000, then the first data Amount is not up to the first data-quantity threshold；And the first data volume of detection be 1,000,000,000, or be greater than 1,000,000,000 when, then it represents that first Data volume has reached the first data-quantity threshold.

In the present invention, primary inspection can be executed to the first data volume in the first memory space at regular intervals It surveys, specifically, is described in detail with embodiments discussed below.

In one preferred embodiment of the invention, above-mentioned steps 205 may include:

Sub-step A: it every prefixed time interval, executes in corresponding first memory space of primary the first index of the detection The first data volume the step of whether reaching the first data-quantity threshold.

Prefixed time interval refers to the time interval of the data volume in corresponding first memory space of the first index of detection, in advance If time interval can be 5h, 1 day, 3 days etc., specifically, can depending on without asking, the embodiment of the present invention to this not It limits.

In the present invention, one-time detection first can be executed every prefixed time interval and indexed in corresponding first memory space The first data volume whether reach the first data-quantity threshold, for example, being carried out every 1 day to the data volume of the first memory space primary Detection, or one-time detection etc. is carried out every 5 hours data volumes to the first memory space.

The behaviour that the embodiment of the present invention passes through the first data volume every set time execution the first memory space of one-time detection Make, can be excessive to avoid the data volume of the first memory space of write-in, cause data that the problem of failing is written.

Index whether the first data volume in corresponding first memory space reaches the first data-quantity threshold in detection first Later, step 205 is executed.

Step 205: in the case where first data volume reaches first data-quantity threshold, creation and described first Index associated second index, and the second memory space corresponding with second index.

Second index refer to creation with first index it is associated one index.

ElasticSearch provides rollover mode, i.e., rotates out the mode newly indexed automatically, in the first index pair In the case that first data volume of the first memory space answered reaches the first data-quantity threshold, it can automatically create and the first index Corresponding second index, that is, re-create an index, and is corresponding second memory space of the second index creation.

It can be the corresponding index mark of the second index setting when creation second indexes, it, can be with when to index management To by under the second index planning to the first index, i.e., second indexes a subindex as the first index, to establish the second rope Draw the incidence relation between the first index, the second index, which can be explicitly indicated out, to be rotated out on the basis of the first index A new index.

It can also be the second index creation one or more fragment when creation second indexes, one or more fragments are For corresponding second memory space of the second index.

Correspondingly, the second memory space also has corresponding second data-quantity threshold, i.e., preset to store with second The setting of the corresponding data-quantity threshold in space, the second data-quantity threshold can be set according to the size of the second memory space, Specifically, it will be described in detail in the following embodiments, the embodiment of the present invention is not repeated here herein.

Step 206: obtaining data to be written.

Data to be written refer to the data of the second memory space to be written.

Data to be written can be operation log data, for example, user's transmission barrage information, can be generated corresponding log Data；Either, user is during blog pushes text, operation log data of generation etc..

Data to be written can also be that the data of other second memory spaces to be written specifically can be according to business need Depending on asking, this is no longer going to repeat them for the embodiment of the present invention.

Data to be written can be in the case where the first data volume of the first memory space reaches the first data-quantity threshold, During the second index creation, data to be written can be constantly obtained, after obtaining data to be written, can first be protected It is stored in spatial cache.

For this process, can be described according to embodiments discussed below.

In one preferred embodiment of the invention, after above-mentioned steps 206, can also include:

Step B: by the data buffer storage to be written in spatial cache, to stop that number is written into first memory space According to.

It in embodiments of the present invention, can be with when the first data volume of the first memory space reaches the first data-quantity threshold Stop that data are written into the first memory space, specifically, reaches the first data-quantity threshold, and the second index in the first data volume Creation during, system still can constantly receive the data of memory space to be written, at this point it is possible to be written into data It is cached in spatial cache, to achieve the purpose that stop that data are written to the first memory space.

And during this, the data to be written for being cached in spatial cache can be in the second index and the creation of the second memory space After completion, the second memory space is written according to the second index.

Certainly, data to be written are also possible to after the second index creation is completed, the data to be written of acquisition, specifically Ground, can according to the actual situation depending on.

It is to be appreciated that the execution sequence of above-mentioned steps 206 and above-mentioned steps 205 is without successive point, step 205 and step 206 can synchronize execution.

Step 207: according to second index, second memory space is written into the data to be written.

Second index is directed toward the second memory space, after obtaining data to be written, can be directed toward according to the second index Second memory space is written into data and the second memory space is written, that is, is written into data and is stored in the second memory space.

It when carrying out write operation to data to be written, can be stored with data separation to be written, i.e., persistent storage is gentle Storage, specifically, is described in conjunction with embodiments discussed below.

In one preferred embodiment of the invention, the second memory space may include the first memory and the second memory, on Stating step 207 may include:

Sub-step C: according to second index, being stored in first memory for the data persistence to be written, and By the data buffer storage to be written in second memory.

In the present invention, the first memory refers to that the memory for persistent storage data to be written, the first memory can be The databases such as mongo.

Second memory refers to that the memory for caching data to be written, the second memory can be the databases such as CouchBase.

After the data to be written for obtaining the second memory space of write-in, needs to be written into data and be respectively written into first Deposit with the second memory, the persistent storage of data to be written may be implemented by the first memory, may be implemented by the second memory The caching of data to be written.

By providing two kinds of data storage methods the backup to write-in data may be implemented, and then deposit at some in the present invention When exception or database corruption occur in the data stored in storage space, it can be obtained from another corresponding memory space Data avoid the loss of data.

The embodiment of the present invention can realize the automatic wound newly indexed when the corresponding memory space data volume of index is larger It builds, also, creates process and be not necessarily to manual intervention, save human cost, realize the purpose of automatic dilatation.

After being written into data and the second memory space is written, step 208 is executed.

Step 208: establishing the second mapping relations between the data to be written and second index, and by described the Two mapping relations are saved to the data routing table.

Second mapping relations refer to the mapping relations between the second index and data to be written.

After the index of acquisition second and data to be written, it can be identified according to the corresponding index of the second index and to be written Enter the corresponding Data Identification of data, establish the second mapping relations between index mark and Data Identification, and this second is mapped Relationship is stored in data routing table, and when subsequent process needs to modify to data to be written, it can be directly according to need The data to be written to be modified obtain second index with mapping relations, and the road according to the second index change replacement data By being directed toward, the modification for completing the data to be written modified needs is operated.

Step 209: when monitoring modification corresponding with target data operation, it is corresponding wait replace to obtain the target data Change data.

It is to be appreciated that target data can be the data having been written into the first memory space, it is also possible to have been written into Data in two memory spaces specifically can be depending on business demand.

Monitoring programme corresponding with operation is modified can be preset in systems, and business personnel's triggering can be monitored in real time Modification operation can be obtained from server and target data pair when monitoring modification corresponding with target data operation The initial data answered in turn modifies to initial data according to modification operation, to obtain the number to be replaced of replacement target data According to.

After obtaining data to be replaced corresponding with target data, step 210 is executed.

Step 210: according to the data routing table being pre-created, searching and indexed with the matched target of the target data；Institute State the mapping relations preserved between write-in data and index in data routing table.

Target index refers to the index with target data there are mapping relations, due to having pre-saved in data routing table The mapping relations between data and index are written, it, can be from the number being pre-created after determining the target data that needs are modified Each of it is indexed according to being searched in routing table with the matched target of target data, for example, index includes index 1 and index 2, have been written into Data both correspond to only one Data Identification, and for distinguishing the different data of write-in, Data Identification can be to be indexed in creation When specified mark, Data Identification includes mark 1~1000 and mark 10001~2000, can be protected in data routing table The mapping relations between the mapping relations between index 1 and mark 1~1000, and index 2 and mark 1001~2000 are deposited, in mesh It is then index 2 with the matched index of Data Identification 2008, then indexing 2 is target rope when the Data Identification for marking data is 2008 Draw.

In the present invention, target index can be for an index in the first index and the second index, i.e. target index The index created in ElasticSearch.

Step 211: being indexed according to the target, the data to be replaced are routed into the target and index corresponding target Memory space.

In the present invention, target storage space is a memory space in the first memory space and the second memory space.

The specific embodiment of above-mentioned steps 211 is similar to the specific embodiment of step 103 in above-described embodiment, this hair Bright embodiment is not repeated here herein.

Step 212: in the target storage space, the target data being replaced with into the data to be replaced.

During following, carried out in conjunction with implementation procedure of the Fig. 3 to the embodiment of the present invention detailed further below.

Referring to Fig. 3, a kind of schematic diagram of data handling procedure provided in an embodiment of the present invention is shown.

As shown in figure 3,1: TV IP address obtains task nIptvtestn, and 2: TV IP address obtains task 2Iptvtest2,3: TV IP address obtain task 1Iptvtest1,4: index node nAPInoden, 5:API (Application Programming Interface, application programming interface) node 1APInode1,6:API node APInode, 7: data source MQ, 8: cache routing table couchBase, 9: persistence routing table MongoDB.

MQ (7) can be used for issuing write-in data to ElasticSearch cluster.

APInode (6), APInode1 (5) ..., APInoden (4) indicate ElasticSearch cluster in API section Point includes multiple servers in ElasticSearch cluster, and each server corresponds to an API node, in current index pair When the data volume for the memory space answered reaches threshold value, all API nodes (i.e. APInode (6), APInode1 can be notified (5) ..., (4) APInoden) stop to currently indexing corresponding memory space write-in data, wherein n is more than or equal to 1 Positive integer.And when ElasticSearch is rotated out and currently indexed associated new index, it can notify all API nodes (i.e. APInode (6), APInode1 (5) ..., APInoden (4)) to newly indexing corresponding memory space write-in data.

It is preserved in couchBase (8) between the data in each index memory space corresponding with each index is stored in Mapping relations.

Reflecting between the data in each index memory space corresponding with each index is stored in is preserved in MongoDB (9) Penetrate relationship.

Since the data saved in couchBase database are generally only caching three months or so, and in couchBase number After the data dump in library, then mapping relations corresponding with clearing data also and then are removed in couchBase (8), are avoided To the occupancy in space.

And the data saved in MongoDB database are that persistence saves, in the write-in data that MongoDB (9) save It is more complete with the mapping relations of index.

When ElasticSearch cluster is received from the data to be written that MQ (7) is issued, can be executed according to current The index of write operation is written into data and writes direct memory space corresponding with the index.

And receive data modification operation when, as in ElasticSearch cluster Iptvtest1 (3), Iptvtest2 (2) ..., Iptvtestn (1), Iptvtest1 (3), Iptvtest2 (2) ..., Iptvtestn (1) respectively It represents differently TV IP address and obtains task, when for Iptvtest1 (3) task, can be appointed according to Iptvtest1 (3) Business obtains corresponding data from MQ (7), and is stored respectively in MongoDB database and couchBase database, and becomes in task When (2) more Iptvtest2, then need to repair the data for having been written into MongoDB database and couchBase database Change, at this point it is possible to inquire MongoDB (9) and couchBase (8), respectively with the number searched with obtained according to Iptvtest2 (2) According to corresponding index, in turn, MongoDB will be respectively written into the data obtained according to Iptvtest2 (2) according to the index of lookup Database and couchBase database ..., when task is changed to (1) Iptvtestn are then needed to having been written into MongoDB The data of database and couchBase database are modified, at this point it is possible to inquire MongoDB (9) and couchBase respectively (8), to search corresponding with the data obtained according to Iptvtestn (21) index, in turn, according to the index of lookup will with according to The data that Iptvtestn (1) is obtained are respectively written into MongoDB database and couchBase database.

The embodiment of the present invention in the follow-up process modifies to written data by creation data routing table When, data routing table can be directly searched, the correct index of data to be replaced corresponding with written data is obtained, according to lookup Index determine that the routing of data to be replaced is directed toward, and complete data replacement, solving in existing program cannot be to having been written into number The problem of according to modification.

Referring to Fig. 4, a kind of structural schematic diagram of data processing equipment provided in an embodiment of the present invention is shown, it specifically can be with Including following module:

Data acquisition module 401 to be replaced, for when monitoring modification corresponding with target data operation, described in acquisition The corresponding data to be replaced of target data；

Target indexes searching module 402, for searching and the target data according to the data routing table being pre-created The target index matched；The mapping relations between write-in data and index are preserved in the data routing table；

Data routing module 403 to be replaced routes to the data to be replaced described for indexing according to the target Target indexes corresponding target storage space；

Target data replacement module 404, for the target data being replaced with described in the target storage space Data to be replaced.

Data processing equipment provided in an embodiment of the present invention, by monitoring modification operation corresponding with target data When, the corresponding data to be replaced of target data are obtained, according to the data routing table being pre-created, are searched matched with target data Target indexes, and the mapping relations between write-in data and index are preserved in data routing table, will be to be replaced according to target index Data route to target and index corresponding target storage space, and in target storage space, target data is replaced with wait replace Change data.The embodiment of the present invention is being saved between write-in data and index in data routing table by creation data routing table Mapping relations, in turn, can need to have been written into modify when, directly according to write-in data and index between mapping Relationship determines that the routing of replacement data is directed toward, and is replaced to corresponding data, it is thus possible to which it is internal to reduce redundant data Deposit the occupancy of resource.

Referring to Fig. 5, a kind of structural schematic diagram of data processing equipment provided in an embodiment of the present invention is shown, it specifically can be with Including following module:

Data routing table creation module 501, for creating the data routing table；

First data acquisition module 502 is indexed for obtaining described first, and is stored in first memory space First data；

First mapping relations establish module 503, first for establishing between first index and first data Mapping relations, and first mapping relations are saved to the data routing table；

First data volume detection module 504, for detecting the first data in corresponding first memory space of the first index Whether amount reaches the first data-quantity threshold；

Second index creation module 505, for the case where first data volume reaches first data-quantity threshold Under, creation and associated second index of first index, and the second memory space corresponding with second index；The mesh Mark index is an index in first index and second index, and the target storage space is first storage A memory space in space and second memory space；

Data acquisition module 506 to be written, for obtaining data to be written；

The data to be written are written described for according to second index by Data write. module 507 to be written Two memory spaces；

Second mapping relations establish module 508, for establishing between the data to be written and second index Two mapping relations, and second mapping relations are saved to the data routing table；

Data acquisition module 509 to be replaced, for when monitoring modification corresponding with target data operation, described in acquisition The corresponding data to be replaced of target data；

Target indexes searching module 510, for searching and the target data according to the data routing table being pre-created The target index matched；The mapping relations between write-in data and index are preserved in the data routing table；

Data routing module 511 to be replaced routes to the data to be replaced described for indexing according to the target Target indexes corresponding target storage space；

Target data replacement module 512, for the target data being replaced with described in the target storage space Data to be replaced.

Preferably, the first data volume detection module 504 includes:

Implementation sub-module 5041 is detected, for primary first data volume being executed and detecting mould every prefixed time interval Block.

Preferably, described device further include:

Data stop writing module 5060, are used for the data buffer storage to be written in spatial cache, to stop to described Data are written in first memory space.

Preferably, second memory space includes the first memory and the second memory, the Data write. module to be written 507 include:

Data sub-module stored 5071 to be written, for being indexed according to described second, by the data persistence to be written It is stored in first memory, and by the data buffer storage to be written in second memory.

For device embodiment, since it is basically similar to the method embodiment, related so being described relatively simple Place illustrates referring to the part of embodiment of the method.

Optionally, the present invention also provides a kind of electronic equipment, comprising: memory, processor and is stored in the memory Computer program that is upper and can running on the processor, the computer program is realized above-mentioned when being executed by the processor Each process of data processing method embodiment, and identical technical effect can be reached, to avoid repeating, which is not described herein again.

Optionally, it the present invention also provides a kind of computer readable storage medium, is deposited on the computer readable storage medium Computer program is contained, the computer program realizes the step in data processing method as described above when being executed by processor Suddenly.Wherein, the computer readable storage medium, such as read-only memory (Read-Only Memory, abbreviation ROM), at random Access memory (Random Access Memory, abbreviation RAM), magnetic or disk etc..

All the embodiments in this specification are described in a progressive manner, the highlights of each of the examples are with The difference of other embodiments, the same or similar parts between the embodiments can be referred to each other.

It should be understood by those skilled in the art that, the embodiment of the embodiment of the present invention can provide as method, apparatus or calculate Machine program product.Therefore, the embodiment of the present invention can be used complete hardware embodiment, complete software embodiment or combine software and The form of the embodiment of hardware aspect.Moreover, the embodiment of the present invention can be used one or more wherein include computer can With in the computer-usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) of program code The form of the computer program product of implementation.

The embodiment of the present invention be referring to according to the method for the embodiment of the present invention, terminal device (system) and computer program The flowchart and/or the block diagram of product describes.It should be understood that flowchart and/or the block diagram can be realized by computer program instructions In each flow and/or block and flowchart and/or the block diagram in process and/or box combination.It can provide these Computer program instructions are set to general purpose computer, special purpose computer, Embedded Processor or other programmable data processing terminals Standby processor is to generate a machine, so that being held by the processor of computer or other programmable data processing terminal devices Capable instruction generates for realizing in one or more flows of the flowchart and/or one or more blocks of the block diagram The device of specified function.

These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing terminal devices In computer-readable memory operate in a specific manner, so that instruction stored in the computer readable memory generates packet The manufacture of command device is included, which realizes in one side of one or more flows of the flowchart and/or block diagram The function of being specified in frame or multiple boxes.

These computer program instructions can also be loaded into computer or other programmable data processing terminal devices, so that Series of operation steps are executed on computer or other programmable terminal equipments to generate computer implemented processing, thus The instruction executed on computer or other programmable terminal equipments is provided for realizing in one or more flows of the flowchart And/or in one or more blocks of the block diagram specify function the step of.

Although the preferred embodiment of the embodiment of the present invention has been described, once a person skilled in the art knows bases This creative concept, then additional changes and modifications can be made to these embodiments.So the following claims are intended to be interpreted as Including preferred embodiment and fall into all change and modification of range of embodiment of the invention.

Finally, it is to be noted that, herein, relational terms such as first and second and the like be used merely to by One entity or operation are distinguished with another entity or operation, without necessarily requiring or implying these entities or operation Between there are any actual relationship or orders.Moreover, the terms "include", "comprise" or its any other variant meaning Covering non-exclusive inclusion, so that process, method, article or terminal device including a series of elements not only wrap Those elements are included, but also including other elements that are not explicitly listed, or further includes for this process, method, article Or the element that terminal device is intrinsic.In the absence of more restrictions, being wanted by what sentence "including a ..." limited Element, it is not excluded that there is also other identical elements in process, method, article or the terminal device for including the element.

Above to a kind of data processing method provided by the present invention, a kind of data processing equipment, a kind of electronic equipment and A kind of computer readable storage medium, is described in detail, and specific case used herein is to the principle of the present invention and reality The mode of applying is expounded, and the above description of the embodiment is only used to help understand the method for the present invention and its core ideas；Together When, for those of ordinary skill in the art, according to the thought of the present invention, have in specific embodiments and applications Change place, in conclusion the contents of this specification are not to be construed as limiting the invention.

Claims

1. a kind of data processing method characterized by comprising

According to the data routing table being pre-created, searches and indexed with the matched target of the target data；The data routing table In preserve write-in data and index between mapping relations；

It is indexed according to the target, the data to be replaced is routed into the target and index corresponding target storage space；

2. the method according to claim 1, wherein obtaining the corresponding number to be replaced of the target data described According to before, further includes:

In the case where first data volume reaches first data-quantity threshold, creation and first index associated the Two indexes, and the second memory space corresponding with second index；Target index is first index and described the An index in two indexes, the target storage space are one in first memory space and second memory space A memory space；

Obtain data to be written；

3. according to the method described in claim 2, it is characterized in that, in corresponding first memory space of the first index of the detection The first data volume whether reach the first data-quantity threshold, comprising:

Every prefixed time interval, the first data volume in corresponding first memory space of primary the first index of the detection is executed The step of whether reaching the first data-quantity threshold.

4. according to the method described in claim 2, it is characterized in that, in corresponding first memory space of first index of detection In the first data volume whether reach the first data-quantity threshold before, further includes:

Create the data routing table；

Obtain first index, and the first data being stored in first memory space；

The first mapping relations between first index and first data are established, and first mapping relations are saved To the data routing table；

Described according to second index, the data to be written are written after second memory space, further includes:

The second mapping relations between the data to be written and second index are established, and second mapping relations are protected It deposits to the data routing table.

5. according to the method described in claim 2, it is characterized in that, after the acquisition data to be written, further includes:

6. according to the method described in claim 2, it is characterized in that, second memory space includes in the first memory and second It deposits, it is described according to second index, second memory space is written into the data to be written, comprising:

According to second index, the data persistence to be written is stored in first memory, and will be described to be written Data buffer storage is in second memory.

7. a kind of data processing equipment characterized by comprising

Data acquisition module to be replaced, for obtaining the number of targets when monitoring modification corresponding with target data operation According to corresponding data to be replaced；

Target indexes searching module, for searching and the matched mesh of the target data according to the data routing table being pre-created Mark index；The mapping relations between write-in data and index are preserved in the data routing table；

The data to be replaced are routed to the target rope for indexing according to the target by data routing module to be replaced Draw corresponding target storage space；

Target data replacement module, for the target data being replaced with described to be replaced in the target storage space Data.

8. device according to claim 7, which is characterized in that described device further include:

First data volume detection module, for detecting whether the first data volume in corresponding first memory space of the first index reaches To the first data-quantity threshold；

Second index creation module, for creating in the case where first data volume reaches first data-quantity threshold With associated second index of first index, and the second memory space corresponding with second index；The target index For it is described first index and it is described second index in one index, the target storage space be first memory space and A memory space in second memory space；

Data acquisition module to be written, for obtaining data to be written；

Data write. module to be written, for according to second index, the data write-in described second to be written to be stored Space.

9. device according to claim 8, which is characterized in that the first data volume detection module includes:

10. device according to claim 8, which is characterized in that described device further include:

Data routing table creation module, for creating the data routing table；

First data acquisition module is indexed for obtaining described first, and the first number being stored in first memory space According to；

First mapping relations establish module, and the first mapping for establishing between first index and first data is closed System, and first mapping relations are saved to the data routing table；

Described device further include:

Second mapping relations establish module, and the second mapping for establishing between the data to be written and second index is closed System, and second mapping relations are saved to the data routing table.

11. device according to claim 8, which is characterized in that described device further include:

Data stop writing module, in spatial cache, storing the data buffer storage to be written to described first with stopping Data are written in space.

12. device according to claim 8, which is characterized in that second memory space includes the first memory and second Memory, the Data write. module to be written include:

Data sub-module stored to be written, for according to second index, the data persistence to be written to be stored in institute The first memory is stated, and by the data buffer storage to be written in second memory.

13. a kind of electronic equipment characterized by comprising memory, processor and be stored on the memory and can be in institute The computer program run on processor is stated, such as claim 1 to 6 is realized when the computer program is executed by the processor Any one of described in data processing method.

14. a kind of computer readable storage medium, which is characterized in that be stored with computer on the computer readable storage medium Program realizes such as data processing side described in any one of claims 1 to 6 when the computer program is executed by processor Method.