CN105335362B - The processing method and system of real time data, instant disposal system for treating - Google Patents
The processing method and system of real time data, instant disposal system for treating Download PDFInfo
- Publication number
- CN105335362B CN105335362B CN201410229319.1A CN201410229319A CN105335362B CN 105335362 B CN105335362 B CN 105335362B CN 201410229319 A CN201410229319 A CN 201410229319A CN 105335362 B CN105335362 B CN 105335362B
- Authority
- CN
- China
- Prior art keywords
- node
- real time
- time data
- treating
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Abstract
The application provides the processing method and system, instant disposal system for treating of a kind of real time data.The embodiment of the present application obtains the real time data that real time processing system is sent by merge node, and then according to distribution policy, in at least one described local node, or in merge node and at least one described local node, determine a node, to enable processing node that the real time data is written in the instant disposal system for treating as processing node, so that realizing instant disposal system for treating flexibly can be calculated or be inquired for real time data.
Description
[technical field]
This application involves the processing methods and system of data processing technique more particularly to a kind of real time data, instant processing
System.
[background technique]
Instant disposal system for treating, can for off-line data flexibly be calculated or be inquired i.e. can not predictable rules meter
It calculates.For example, the male to female ratio of link integrity member and non-link integrity member in member are calculated, alternatively, for another example calculating in member really
The transaction medal ratio of logical member and non-link integrity member are believed, alternatively, for another example calculating link integrity member and non-sincerity in member
The ratio of certification on the spot etc. of logical member.
However, existing instant disposal system for treating flexibly can not be calculated or be inquired for real time data.
[summary of the invention]
The many aspects of the application provide the processing method and system, instant disposal system for treating of a kind of real time data, to reality
Existing instant disposal system for treating flexibly can be calculated or be inquired for real time data, especially can be adapted for counting magnanimity in real time
According to calculating or inquiry.
The one side of the application provides a kind of processing method of real time data, is applied in instant disposal system for treating, described to be
When processing system include merge node and at least one local node, which comprises
The merge node obtains the real time data that real time processing system is sent；
The merge node determines a node as processing node, to by the real time data according to distribution policy
It is written in the instant disposal system for treating；Wherein,
The processing node includes a local node in the merge node or at least one described local node.
The aspect and any possible implementation manners as described above, it is further provided a kind of implementation, the merging section
Point according to distribution policy, determine a node, using as processing node after, further includes:
The real time data is written in the instant disposal system for treating processing node.
The aspect and any possible implementation manners as described above, it is further provided a kind of implementation, the processing section
The real time data is written in the instant disposal system for treating point, comprising:
The processing node creates the full-text index file of the real time data；
The full-text index file is written in the instant disposal system for treating processing node.
The aspect and any possible implementation manners as described above, it is further provided a kind of implementation, the processing section
The full-text index file is written in the instant disposal system for treating point, comprising:
The status condition of real time data described in the processing monitoring nodes；
If the status condition meets the first Writing condition, the processing node will be described in full-text index file write-in
In the speedy storage equipment of instant disposal system for treating.
The aspect and any possible implementation manners as described above, it is further provided a kind of implementation, if described
Status condition meets the first Writing condition, and the instant disposal system for treating is written in the full-text index file by the processing node
In speedy storage equipment, comprising:
If the receiving time of the real time data reaches the first maximum SEE time, alternatively, the quantity of the real time data
Reach the first maximum number of files, quickly depositing for the instant disposal system for treating is written in the full-text index file by the processing node
It stores up in equipment.
The aspect and any possible implementation manners as described above, it is further provided a kind of implementation, the processing section
The full-text index file is written in the instant disposal system for treating point, further includes:
If the status condition meets the second Writing condition, the processing node will be described in full-text index file write-in
The storage equipment of instant disposal system for treating is written in the slow storage device of the instant disposal system for treating.
The aspect and any possible implementation manners as described above, it is further provided a kind of implementation, if described
Status condition meets the second Writing condition, and the instant disposal system for treating is written in the full-text index file by the processing node
In slow storage device, comprising:
If the receiving time of the real time data reaches the second maximum SEE time, alternatively, the quantity of the real time data
Reach the second maximum number of files, depositing at a slow speed for the instant disposal system for treating is written in the full-text index file by the processing node
Store up equipment.
The aspect and any possible implementation manners as described above, it is further provided a kind of implementation, the processing section
Point monitors after the status condition of the real time data, further includes:
If the status condition meets at least one in first Writing condition and second Writing condition, described
Processing node starts new query engine, to carry out the immediate inquiring of the real time data.
The aspect and any possible implementation manners as described above, it is further provided a kind of implementation, the processing section
It puts and starts new query engine, after the immediate inquiring to carry out the real time data, further includes:
The merge node receives data inquiry request, includes querying condition in the data inquiry request；
Data inquiry request is distributed at least one described local node by the merge node；
Each local in each local node or at least one described local node at least one described local node
Node and the merge node execute calculating operation corresponding to querying condition respectively, to obtain query result, and return to institute
State merge node；
The merge node merges the query result, to obtain final query result.
The another aspect of the application provides a kind of instant disposal system for treating, including merge node and at least one local node,
Wherein,
The merge node, for obtaining the real time data of real time processing system transmission；And it according to distribution policy, determines
One processing node；Wherein, the processing node includes one in the merge node or at least one described local node
Local node；
The processing node, for the real time data to be written in the instant disposal system for treating.
The aspect and any possible implementation manners as described above, it is further provided a kind of implementation, the merging section
Point and at least one described local node form distributed cloud framework.
The aspect and any possible implementation manners as described above, it is further provided a kind of implementation, the processing section
Point, is specifically used for
Create the full-text index file of the real time data；
The full-text index file is written in the instant disposal system for treating.
The aspect and any possible implementation manners as described above, it is further provided a kind of implementation, the processing section
Point, is specifically used for
Monitor the status condition of the real time data；And
If the status condition meets the first Writing condition, the instant disposal system for treating is written into the full-text index file
Speedy storage equipment in.
The aspect and any possible implementation manners as described above, it is further provided a kind of implementation, the processing section
Point, is specifically used for
If the receiving time of the real time data reaches the first maximum SEE time, alternatively, the quantity of the real time data
Reach the first maximum number of files, quickly depositing for the instant disposal system for treating is written in the full-text index file by the processing node
It stores up in equipment.
The aspect and any possible implementation manners as described above, it is further provided a kind of implementation, the processing section
Point, is specifically used for
If the status condition meets the second Writing condition, the instant disposal system for treating is written into the full-text index file
Storage equipment be written in the slow storage device of the instant disposal system for treating.
The aspect and any possible implementation manners as described above, it is further provided a kind of implementation, the processing section
Point, is specifically used for
If the receiving time of the real time data reaches the second maximum SEE time, alternatively, the quantity of the real time data
Reach the second maximum number of files, depositing at a slow speed for the instant disposal system for treating is written in the full-text index file by the processing node
Store up equipment.
The aspect and any possible implementation manners as described above, it is further provided a kind of implementation, the processing section
Point, is also used to
If the status condition meets at least one in first Writing condition and second Writing condition, starting
New query engine, to carry out the immediate inquiring of the real time data.
The aspect and any possible implementation manners as described above, it is further provided a kind of implementation, the merging section
Point, is also used to
Data inquiry request is received, includes querying condition in the data inquiry request；
Data inquiry request is distributed at least one described local node, so that at least one described local node
Each local node and the merge node in each local node or at least one described local node execute look into respectively
Calculating operation corresponding to inquiry condition, to obtain query result；
Obtain the query result；
The query result is merged, to obtain final query result.
The another aspect of the application provides a kind of processing system of real time data, including real time processing system and above-mentioned one
Instant disposal system for treating provided by aspect；Wherein,
The real time processing system, for sending the real time data to the instant disposal system for treating.
As shown from the above technical solution, the embodiment of the present application obtains the real-time of real time processing system transmission by merge node
Data, and then according to distribution policy, at least one described local node or merge node and at least one described local
In node, a node is determined, to enable processing node that real time data write-in is described immediately as processing node
In processing system, so that realizing instant disposal system for treating flexibly can be calculated or be inquired for real time data.
In addition, using technical solution provided by the present application, since instant disposal system for treating can carry out spirit for real time data
Work is calculated or inquiry, therefore, can effectively improve the flexibility of real time data immediate inquiring.
In addition, using technical solution provided by the present application, due to creating the full-text index file of real time data, and by institute
It states in full-text index file write-in instant disposal system for treating, allows to be counted in real time using the full-text index file of real time data
According to immediate inquiring, without being inquired in entire real time data, therefore, the system that can reduce instant disposal system for treating
Expense, to improve the efficiency of real time data immediate inquiring.
[Detailed description of the invention]
It in order to more clearly explain the technical solutions in the embodiments of the present application, below will be to embodiment or description of the prior art
Needed in attached drawing be briefly described, it should be apparent that, the accompanying drawings in the following description is some realities of the application
Example is applied, it for those of ordinary skill in the art, without any creative labor, can also be attached according to these
Figure obtains other attached drawings.
Fig. 1 is the flow diagram of the processing method for the real time data that one embodiment of the application provides；
Fig. 2 is the structural schematic diagram for the instant disposal system for treating that another embodiment of the application provides；
Fig. 3 is the structural schematic diagram of the processing system for the real time data that another embodiment of the application provides.
[specific embodiment]
To keep the purposes, technical schemes and advantages of the embodiment of the present application clearer, below in conjunction with the embodiment of the present application
In attached drawing, the technical scheme in the embodiment of the application is clearly and completely described, it is clear that described embodiment is
Some embodiments of the present application, instead of all the embodiments.Based on the embodiment in the application, those of ordinary skill in the art
Whole other embodiments obtained without creative efforts, shall fall in the protection scope of this application.
In addition, the terms "and/or", only a kind of incidence relation for describing affiliated partner, indicates may exist
Three kinds of relationships, for example, A and/or B, can indicate: individualism A exists simultaneously A and B, these three situations of individualism B.Separately
Outside, character "/" herein typicallys represent the relationship that forward-backward correlation object is a kind of "or".
Fig. 1 is the flow diagram of the processing method for the real time data that one embodiment of the application provides, and is applied to place immediately
In reason system, as shown in Fig. 2, the instant disposal system for treating includes merge node and at least one local node, as shown in Figure 1.
101, the merge node obtains the real time data that real time processing system is sent.
Optionally, in a possible implementation of the present embodiment, the received real time data of the merge node institute
It can be flow data (Streaming Data) caused by real time processing system institute directly received real time data source, for example,
Network monitoring system, banking analysis system, traffic flow forecasting system, WWW (World Wide Web, referred to as Web) etc.
Flow data caused by application system, or can also be real time processing system to stream caused by received real time data source
Data are calculated according to unalterable rules, such as summation etc., and calculated result obtained, the present embodiment is to this without special
It limits.
Specifically, real time processing system can be by calling the data of instant disposal system for treating to update application programming interfaces
(Application Programming Interface, API) is sent out by the request processing component reception of instant disposal system for treating
The data write request sent includes real time data in the data write request.Request processing component asks data write-in
It asks and is sent to merge node, be responsible for handling the data write request by the merge node.
Wherein, the merge node can be a preconfigured stationary nodes, or can also be request processing group
The randomly selected local node of part, or can also be the local section that request processing component is selected according to election strategy
Point, the present embodiment is to this without being particularly limited to.
102, the merge node is according to distribution policy, determines a node as processing node, to will be described real-time
Data are written in the instant disposal system for treating.
Wherein, the processing node includes a local section in the merge node or at least one described local node
Point.
Optionally, in a possible implementation of the present embodiment, if the merge node is preconfigured
One stationary nodes, then, in 102, the merge node can then be saved according to distribution policy from least one described local
In point, a node is determined, using as the processing node.At this point, the processing node can be then at least one described sheet
A local node in ground node.
Optionally, in a possible implementation of the present embodiment, if the merge node is request processing group
The local node that the randomly selected local node of part or request processing component are selected according to election strategy, that
, in 102, the merge node can then be saved according to distribution policy from the merge node and at least one described local
In point, a node is determined, using as the processing node.At this point, the processing node then can for the merge node or
A local node at least one described local node.
Specifically, the distribution policy can include but is not limited to Hash operation strategy and polling schemas, the present embodiment pair
This is without being particularly limited to.
For example, the merge node can identification information (ID) to real time data carry out Hash operation, determine one it is right
The node answered, using as processing node；Alternatively, for another example the merge node can also sequentially select one using polling schemas
A node, using as processing node；The present embodiment is to this without being particularly limited to.
Optionally, in a possible implementation of the present embodiment, after 102, the processing node will be described
Real time data is written in the instant disposal system for treating.
Specifically, merge node can write data into request and consign to processing node.It is understood that if described
Merge node is a preconfigured stationary nodes, and the merge node can specifically write data into request and be transmitted to processing
Node；If the merge node is the randomly selected local node of request processing component or request processing component
According to the local node that election strategy selects, the merge node specifically can be forwarded no longer, directly be asked to data write-in
Ask carry out subsequent operation.
After receiving the data write request, the processing node can specifically create the full text of the real time data
Index file.Then, the processing node full-text index file can then be written in the instant disposal system for treating.Specifically
The detailed description on ground, creation full-text index document method may refer to related content in the prior art, and details are not described herein again.
In this way, since processing node creates the full-text index file of real time data, and the full-text index file is write
Enter in instant disposal system for treating, allows to carry out the immediate inquiring of real time data using the full-text index file of real time data, and
Without being inquired in entire real time data, therefore, the overhead of instant disposal system for treating can reduce, to improve
The efficiency of real time data immediate inquiring.
So-called herein " processing in real time ", also known as Stream Processing, that is, process inquiry and calculating, refer to the operation applied at any time
It is handled at any time.For example, can calculate at any time in large-scale double 11 activities for selling website and end current turnover.In Ali
The transaction live streaming Room is named in Ba Ba, this application.Data flow is constantly generated with the time, and often carrying out batch of data stream, (its time interval can
To be set as second grade, minute grade etc. according to application), it just calculates primary.
So-called herein " processing immediately " refers to after application operation, is handled in very short time.For example, user sends out
Primary request out, application internal calculation is completed in very short time, is then returned the result.
It, can be immediately by the full text after the processing node creation finishes the full-text index file of the real time data
Index file is written in the instant disposal system for treating, or can not also be described immediately by full-text index file write-in immediately
In processing system, but met certain condition to the real time data and then full-text index file write-in is described i.e.
When processing system in.In this way, the overhead of instant disposal system for treating can be effectively reduced.
For example, the processing node specifically can selectively be held by the status condition of the monitoring real time data
It is about to the operation that the full-text index file is written in the instant disposal system for treating.
For example, if the status condition meets the first Writing condition, for example, the receiving time of the real time data reaches the
One maximum SEE time, that is, maxTime1, alternatively, for another example the quantity of the real time data reaches the first maximum number of files i.e.
The quick storage of the instant disposal system for treating can be then written in the full-text index file by maxDoc1 etc., the processing node
Equipment, for example, in memory.Wherein, the memory of the instant disposal system for treating can be the memory of computer, or can also be mobile phone
Running memory, that is, Installed System Memory, for example, random access memory (Random Access Memory, RAM) etc., the present embodiment is to this
Without limiting.In some cases, the status condition of the real time data meets the first Writing condition and the full text rope that executes
Soft submission operation can also be referred to as by drawing file write operations.
Alternatively, for another example if the status condition meets the second Writing condition, for example, when the reception of the real time data
Between reach the second maximum SEE time i.e. maxTime2, alternatively, for another example the quantity of the real time data reaches the second maximum text
Then the slow of the instant disposal system for treating can be written in the full-text index file by gear number, that is, maxDoc2 etc., the processing node
Speed storage equipment, for example, hard disk, or can also be inoperative memory, that is, physical memory of mobile phone, for example, read-only memory
(Read-Only Memory, ROM) and RAM card etc., the present embodiment is to this without limiting.In some cases, described real-time
The full-text index file write operations that the status condition of data meets the first Writing condition and executes can also be submitted referred to as firmly
Operation.
Alternatively, for another example if the status condition meets in first Writing condition and second Writing condition
At least one of, for example, first Writing condition can reach for the receiving time of the real time data first it is maximum visible when
Between i.e. maxTime1, alternatively, for another example the quantity of the real time data reaches second maximum number of files i.e. maxDoc1 etc.；Example
Such as, second Writing condition can reach the first maximum SEE time i.e. for the receiving time of the real time data
MaxTime2, alternatively, for another example the quantity of the real time data reaches second maximum number of files i.e. maxDoc2 etc., the processing
Node then can further start new query engine, to carry out the immediate inquiring of the real time data.
It is understood that the first Writing condition and the second Writing condition do not have any relationship, the present embodiment to this not into
Row is particularly limited to.
So far, instant disposal system for treating can be realized by real time data in the form of full-text index file, and processing immediately is written
In system, to ensure that the immediate inquiring of real time data.
In this way, application system program can be by the inquiry API of calling instant disposal system for treating, by asking for instant disposal system for treating
It asks processing component to receive transmitted data inquiry request, includes querying condition in the data inquiry request.Request processing group
The data inquiry request is sent to merge node by part, is responsible for handling the data inquiry request by the merge node.
Specifically, data inquiry request can be distributed at least one described local node by merge node.It is understood that
If the merge node is a preconfigured stationary nodes, the merge node specifically can be by data query
Request is distributed at least one described local node, which no longer executes subsequent operation to the data inquiry request；Such as
Merge node described in fruit is the randomly selected local node of request processing component or request processing component according to election
Data inquiry request can be specifically distributed at least one described sheet by one local node of policy selection, the merge node
While ground node, which continues to execute subsequent operation to the data inquiry request.
After receiving the data inquiry request, each local node at least one described local node, Huo Zhesuo
Each local node and the merge node at least one local node are stated, it is right to be inquired i.e. execution querying condition institute respectively
The calculating operation answered to obtain query result, and returns to merge node, and then is closed query result by merge node
And to obtain final query result, and return to application system.
So far, instant disposal system for treating for real time data be finished one-time efficacious work calculate or inquiry.
It should be noted that the merge node and at least one described local node form distribution in the present embodiment
Cloud framework.It specifically can be by configuration management center for example, ZooKeeper etc., carries out the information configuration and management of centralization, energy
Enough realize the automatic load balancing of automatic fault tolerant and request.For example, each node, that is, merge node and at least one described local
Each local node in node may include a primary node and at least one standby node.If primary node breaks down
When unavailable, a standby node can be used as primary node, continue offer service, to realize automatic fault tolerant.Each spare section
Point can have an opportunity to become primary node, provide service according to balance policy, to realize the automatic load balancing of request.
With the development of information-intensive society, more and more information are by digitization, especially along with the development of internet, number
According to being in explosive growth, there is a large amount of real time data, magnanimity real time data can be referred to as.Due to using by merge node
It is formed by distributed cloud framework at least one local node, and combines technical solution provided by the invention, it can be fine
Ground carries out immediate inquiring or calculating to this magnanimity real time data.
In the present embodiment, the real time data that real time processing system is sent is obtained by merge node, and then according to distribution plan
Slightly, at least one described local node or in merge node and at least one described local node, a section is determined
Point, to enable processing node that the real time data is written in the instant disposal system for treating as processing node, thus real
Having showed instant disposal system for treating flexibly can be calculated or be inquired for real time data.
In addition, using technical solution provided by the present application, since instant disposal system for treating can carry out spirit for real time data
Work is calculated or inquiry, therefore, can effectively improve the flexibility of real time data immediate inquiring.
In addition, using technical solution provided by the present application, due to creating the full-text index file of real time data, and by institute
It states in full-text index file write-in instant disposal system for treating, allows to be counted in real time using the full-text index file of real time data
According to immediate inquiring, without being inquired in entire real time data, therefore, the system that can reduce instant disposal system for treating
Expense, to improve the efficiency of real time data immediate inquiring.
It should be noted that for the various method embodiments described above, for simple description, therefore, it is stated as a series of
Combination of actions, but those skilled in the art should understand that, the application is not limited by the described action sequence because
According to the application, some steps may be performed in other sequences or simultaneously.Secondly, those skilled in the art should also know
It knows, the embodiments described in the specification are all preferred embodiments, related actions and modules not necessarily the application
It is necessary.
In the above-described embodiments, it all emphasizes particularly on different fields to the description of each embodiment, there is no the portion being described in detail in some embodiment
Point, reference can be made to the related descriptions of other embodiments.
Fig. 2 is the structural schematic diagram for the instant disposal system for treating that another embodiment of the application provides, as shown in Figure 2.This implementation
The instant disposal system for treating of example may include merge node 21 and at least one local node 221、222……22n.Wherein, n is big
In or equal to 1 integer.The merge node 21 and the local node 221、222……22nBy wired or wireless network into
Row connection.Merge node 21 and at least one local node 221、222……22n, these nodes one cluster of composition, the cluster
It can be made of 1 or multiple servers, each node occupies the process resource of a part of cluster, that is to say, that each node
A server is corresponded to, multiple nodes can be disposed on every server.
Wherein,
The merge node 21, for obtaining the real time data of real time processing system transmission；And according to distribution policy, really
A fixed processing node；Wherein, the processing node includes the merge node 21 or at least one described local node 221、
222……22nIn a local node 22i, i is the integer greater than 0, and less than or equal to n.
The processing node, for the real time data to be written in the instant disposal system for treating.
Optionally, in a possible implementation of the present embodiment, the 21 received real-time number of institute of merge node
According to can be real time processing system institute directly flow data (Streaming Data) caused by received real time data source, example
Such as, flow data caused by the application systems such as network monitoring system, banking analysis system, traffic flow forecasting system, Web, or
Person can also be real time processing system to flow data caused by received real time data source, be counted according to unalterable rules
It calculates, such as summation etc., calculated result obtained, the present embodiment is to this without being particularly limited to.
Specifically, real time processing system 25 can be by calling the data of instant disposal system for treating to update application programming interfaces
(Application Programming Interface, API) 23 receives institute by the request processing component of instant disposal system for treating
The data write request of transmission includes real time data in the data write request.The data are written request processing component
Request is sent to merge node 21, is responsible for the processing data write request by the merge node 21.
Wherein, the merge node 21 can be a preconfigured stationary nodes, or can also be that request is handled
The randomly selected local node 22 of componentj, j is the integer greater than 0, and less than or equal to n, or be can also be at request
The local node 22 that reason component is selected according to election strategyj, j is the integer greater than 0, and less than or equal to n, the present embodiment
To this without being particularly limited to.
Optionally, in a possible implementation of the present embodiment, if the merge node 21 is to be pre-configured with
A stationary nodes, then, the merge node 21 then can be according to distribution policy, from least one described local node
221、222……22nIn, a node is determined, using as the processing node.At this point, the processing node then can be described
At least one local node 221、222……22nIn a local node 22i, i be greater than 0, and it is whole less than or equal to n
Number.
Optionally, in a possible implementation of the present embodiment, if the merge node 21 is request processing
The randomly selected local node 22 of componentjOr the local section that request processing component is selected according to election strategy
Point 22j, then, the merge node 21 then can be according to distribution policy, and the merge node 21 is according to distribution policy, from described
Merge node 21 and at least one described local node 221、222……22nIn, a node is determined, to save as the processing
Point.At this point, the processing node then can be the merge node 21 or at least one described local node 221、222……22n
In a local node 22i。
Specifically, the distribution policy can include but is not limited to Hash operation strategy and polling schemas, the present embodiment pair
This is without being particularly limited to.
For example, the merge node 21 can identification information (ID) to real time data carry out Hash operation, determine one
Corresponding node, using as processing node；Alternatively, for another example the merge node 21 can also sequentially be selected using polling schemas
Select a node, using as processing node；The present embodiment is to this without being particularly limited to.
Specifically, merge node 21 can write data into request and consign to processing node.It is understood that if institute
Stating merge node 21 is a preconfigured stationary nodes, and the merge node 21 can specifically write data into request forwarding
Give processing node；If the merge node 21 is the randomly selected local node 22 of request processing componentj, or ask
A local node 22 for asking processing component to be selected according to election strategyj, the merge node 21 specifically can forward no longer, directly
It connects and subsequent operation is carried out to data write request.
After receiving the data write request, the processing node specifically can be used for creating the real time data
Full-text index file；And the full-text index file is written in the instant disposal system for treating.Specifically, full-text index is created
The detailed description of document method may refer to related content in the prior art, and details are not described herein again.
In this way, since processing node creates the full-text index file of real time data, and the full-text index file is write
Enter in instant disposal system for treating, allows to carry out the immediate inquiring of real time data using the full-text index file of real time data, and
Without being inquired in entire real time data, therefore, the overhead of instant disposal system for treating can reduce, to improve
The efficiency of real time data immediate inquiring.
So-called herein " processing in real time ", also known as Stream Processing, that is, process inquiry and calculating, refer to the operation applied at any time
It is handled at any time.For example, can calculate at any time in large-scale double 11 activities for selling website and end current turnover.In Ali
The transaction live streaming Room is named in Ba Ba, this application.Data flow is constantly generated with the time, and often carrying out batch of data stream, (its time interval can
To be set as second grade, minute grade etc. according to application), it just calculates primary.
So-called herein " processing immediately " refers to after application operation, is handled in very short time.For example, user sends out
Primary request out, application internal calculation is completed in very short time, is then returned the result.
It, can be immediately by the full text after the processing node creation finishes the full-text index file of the real time data
Index file is written in the instant disposal system for treating, or can not also be described immediately by full-text index file write-in immediately
In processing system, but met certain condition to the real time data and then full-text index file write-in is described i.e.
When processing system in.In this way, the overhead of instant disposal system for treating can be effectively reduced.
For example, the processing node specifically can selectively be held by the status condition of the monitoring real time data
It is about to the operation that the full-text index file is written in the instant disposal system for treating.
For example, if the status condition meets the first Writing condition, for example, the receiving time of the real time data reaches the
One maximum SEE time, that is, maxTime1, alternatively, for another example the quantity of the real time data reaches the first maximum number of files i.e.
The quick storage of the instant disposal system for treating can be then written in the full-text index file by maxDoc1 etc., the processing node
Equipment, for example, in memory.Wherein, the memory of the instant disposal system for treating can be the memory of computer, or can also be mobile phone
Running memory, that is, Installed System Memory, for example, random access memory (Random Access Memory, RAM) etc., the present embodiment is to this
Without limiting.In some cases, the status condition of the real time data meets the first Writing condition and the full text rope that executes
Soft submission operation can also be referred to as by drawing file write operations.
Alternatively, for another example if the status condition meets the second Writing condition, for example, when the reception of the real time data
Between reach the second maximum SEE time i.e. maxTime2, alternatively, for another example the quantity of the real time data reaches the second maximum text
Then the slow of the instant disposal system for treating can be written in the full-text index file by gear number, that is, maxDoc2 etc., the processing node
Speed storage equipment, for example, hard disk, or can also be inoperative memory, that is, physical memory of mobile phone, for example, read-only memory
(Read-Only Memory, ROM) and RAM card etc., the present embodiment is to this without limiting.In some cases, described real-time
The full-text index file write operations that the status condition of data meets the first Writing condition and executes can also be submitted referred to as firmly
Operation.
Alternatively, for another example if the status condition meets in first Writing condition and second Writing condition
At least one of, for example, first Writing condition can reach for the receiving time of the real time data first it is maximum visible when
Between i.e. maxTime1, alternatively, for another example the quantity of the real time data reaches second maximum number of files i.e. maxDoc1 etc.；Example
Such as, second Writing condition can reach the first maximum SEE time i.e. for the receiving time of the real time data
MaxTime2, alternatively, for another example the quantity of the real time data reaches second maximum number of files i.e. maxDoc2 etc., the processing
Node then can further start new query engine, to carry out the immediate inquiring of the real time data.
It is understood that the first Writing condition and the second Writing condition do not have any relationship, the present embodiment to this not into
Row is particularly limited to.
So far, instant disposal system for treating can be realized by real time data in the form of full-text index file, and processing immediately is written
In system, to ensure that the immediate inquiring of real time data.
In this way, application system program 26 can be by the inquiry API24 of calling instant disposal system for treating, by instant disposal system for treating
Request processing component receive transmitted data inquiry request, include querying condition in the data inquiry request.At request
It manages component and the data inquiry request is sent to merge node 21, the processing data query is responsible for by the merge node 21
Request.
Specifically, data inquiry request can be distributed at least one described local node 22 by merge node 211、
222……22n.It is understood that if the merge node 21 is a preconfigured stationary nodes, the merging section
Data inquiry request can be specifically distributed at least one described local node 22 by point 211、222……22n, the merge node
21 no longer execute subsequent operation to the data inquiry request；If the merge node 21 is that request processing component is randomly selected
One local node 22jOr the local node 22 that request processing component is selected according to election strategyj, the merging section
Data inquiry request can be specifically distributed at least one described local node 22 by point 211、222……22nWhile, the conjunction
And node 21 continues to execute subsequent operation to the data inquiry request.
After receiving the data inquiry request, at least one described local node 221、222……22nIn each
Ground node or at least one described local node 221、222……22nIn each local node and the merge node 21, point
It is not inquired and executes calculating operation corresponding to querying condition, to obtain query result, and return to merge node 21, into
And merged query result by merge node 21, to obtain final query result, and return to application system.
So far, instant disposal system for treating for real time data be finished one-time efficacious work calculate or inquiry.
It should be noted that the merge node and at least one described local node form distribution in the present embodiment
Cloud framework.It specifically can be by configuration management center for example, ZooKeeper etc., carries out the information configuration and management of centralization, energy
Enough realize the automatic load balancing of automatic fault tolerant and request.For example, each node, that is, merge node 21 and at least one described sheet
Ground node 221、222……22nIn each local node, may include a primary node and at least one standby node.If main
When breaking down unavailable with node, a standby node can be used as primary node, continue offer service, to realize from visibly moved
It is wrong.Each standby node can have an opportunity to become primary node, provide service according to balance policy, to realize oneself requested
Dynamic load is balanced.
With the development of information-intensive society, more and more information are by digitization, especially along with the development of internet, number
According to being in explosive growth, there is a large amount of real time data, magnanimity real time data can be referred to as.Due to using by merge node
It is formed by distributed cloud framework at least one local node, and combines technical solution provided by the invention, it can be fine
This magnanimity real time data of ground processing.
In the present embodiment, the real time data that real time processing system is sent is obtained by merge node, and then according to distribution plan
Slightly, at least one described local node or in merge node and at least one described local node, a section is determined
Point, to enable processing node that the real time data is written in the instant disposal system for treating as processing node, thus real
Having showed instant disposal system for treating flexibly can be calculated or be inquired for real time data.
In addition, using technical solution provided by the present application, since instant disposal system for treating can carry out spirit for real time data
Work is calculated or inquiry, therefore, can effectively improve the flexibility of real time data immediate inquiring.
In addition, using technical solution provided by the present application, due to creating the full-text index file of real time data, and by institute
It states in full-text index file write-in instant disposal system for treating, allows to be counted in real time using the full-text index file of real time data
According to immediate inquiring, without being inquired in entire real time data, therefore, the system that can reduce instant disposal system for treating
Expense, to improve the efficiency of real time data immediate inquiring.
Fig. 3 is the structural schematic diagram of the processing system for the real time data that another embodiment of the application provides, as shown in Figure 3.
The processing system of the real time data of the present embodiment may include that the embodiment corresponding with Fig. 2 of real time processing system 31 is provided i.e.
When processing system 32.Wherein,
The real time processing system 31, for sending the real time data to the instant disposal system for treating 32.
The detailed description of the instant disposal system for treating 32 may refer to the related content in the corresponding embodiment of Fig. 2, herein
It repeats no more.
In the present embodiment, the real time data that real time processing system is sent is obtained by merge node, and then according to distribution plan
Slightly, at least one described local node or in merge node and at least one described local node, a section is determined
Point, to enable processing node that the real time data is written in the instant disposal system for treating as processing node, thus real
Having showed instant disposal system for treating flexibly can be calculated or be inquired for real time data.
In addition, using technical solution provided by the present application, since instant disposal system for treating can carry out spirit for real time data
Work is calculated or inquiry, therefore, can effectively improve the flexibility of real time data immediate inquiring.
In addition, using technical solution provided by the present application, due to creating the full-text index file of real time data, and by institute
It states in full-text index file write-in instant disposal system for treating, allows to be counted in real time using the full-text index file of real time data
According to immediate inquiring, without being inquired in entire real time data, therefore, the system that can reduce instant disposal system for treating
Expense, to improve the efficiency of real time data immediate inquiring.
It is apparent to those skilled in the art that for convenience and simplicity of description, the system of foregoing description,
The specific work process of device and unit, can refer to corresponding processes in the foregoing method embodiment, and details are not described herein.
In several embodiments provided herein, it should be understood that disclosed system, device and method can be with
It realizes by another way.For example, the apparatus embodiments described above are merely exemplary, for example, the unit
It divides, only a kind of logical function partition, there may be another division manner in actual implementation, such as multiple units or the page
Component can be combined or can be integrated into another system, or some features can be ignored or not executed.Another point is shown
The mutual coupling, direct-coupling or communication connection shown or discussed can be through some interfaces, between device or unit
Coupling or communication connection are connect, can be electrical property, mechanical or other forms.
The unit as illustrated by the separation member may or may not be physically separated, aobvious as unit
The component shown may or may not be physical unit, it can and it is in one place, or may be distributed over multiple
In network unit.It can select some or all of unit therein according to the actual needs to realize the mesh of this embodiment scheme
's.
It, can also be in addition, each functional unit in each embodiment of the application can integrate in one processing unit
It is that each unit physically exists alone, can also be integrated in one unit with two or more units.Above-mentioned integrated list
Member both can take the form of hardware realization, can also realize in the form of hardware adds SFU software functional unit.
The above-mentioned integrated unit being realized in the form of SFU software functional unit can store and computer-readable deposit at one
In storage media.Above-mentioned SFU software functional unit is stored in a storage medium, including some instructions are used so that a computer
It is each that equipment (can be personal computer, server or the network equipment etc.) or processor (processor) execute the application
The part steps of embodiment the method.And storage medium above-mentioned includes: USB flash disk, mobile hard disk, read-only memory (Read-
Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic or disk etc. it is various
It can store the medium of program code.
Finally, it should be noted that above embodiments are only to illustrate the technical solution of the application, rather than its limitations；Although
The application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it still may be used
To modify the technical solutions described in the foregoing embodiments or equivalent replacement of some of the technical features；
And these are modified or replaceed, each embodiment technical solution of the application that it does not separate the essence of the corresponding technical solution spirit and
Range.
Claims (16)
1. a kind of processing method of real time data, which is characterized in that be applied in instant disposal system for treating, the instant disposal system for treating
Including merge node and at least one local node, which comprises
The merge node obtains the real time data that real time processing system is sent；
The merge node determines a node as processing node according to distribution policy；
The processing node creates the full-text index file of the real time data, and monitors the status condition of the real time data；
If the status condition meets the second Writing condition, the processing node is described immediately by full-text index file write-in
In the slow storage device of processing system；Wherein,
The processing node includes a local node in the merge node or at least one described local node.
2. the method according to claim 1, wherein this method further include:
If the status condition meets the first Writing condition, the processing node is described immediately by full-text index file write-in
In the speedy storage equipment of processing system.
If 3. according to the method described in claim 2, it is characterized in that, the status condition meet the first Writing condition,
The full-text index file is written in the speedy storage equipment of the instant disposal system for treating processing node, comprising:
If the receiving time of the real time data reaches the first maximum SEE time, alternatively, the quantity of the real time data reaches
First maximum number of files, the processing node set the quick storage that the instant disposal system for treating is written in the full-text index file
In standby.
If 4. the method according to claim 1, wherein the status condition meet the second Writing condition,
The full-text index file is written in the slow storage device of the instant disposal system for treating processing node, comprising:
If the receiving time of the real time data reaches the second maximum SEE time, alternatively, the quantity of the real time data reaches
Second maximum number of files, the processing node set the storage at a slow speed that the instant disposal system for treating is written in the full-text index file
In standby.
5. according to the method described in claim 2, it is characterized in that, the status bar of real time data described in the processing monitoring nodes
After part, further includes:
If the status condition meets first Writing condition, the processing node starts new query engine, to carry out
The immediate inquiring of the real time data.
6. the method according to claim 1, wherein the status bar of real time data described in the processing monitoring nodes
After part, further includes:
If the status condition meets second Writing condition, the processing node starts new query engine, to carry out
The immediate inquiring of the real time data.
7. method according to claim 5 or 6, which is characterized in that the processing node starts new query engine, to
After the immediate inquiring for carrying out the real time data, further includes:
The merge node receives data inquiry request, includes querying condition in the data inquiry request；
Data inquiry request is distributed at least one described local node by the merge node；
Each local node in each local node or at least one described local node at least one described local node
With the merge node, calculating operation corresponding to querying condition is executed respectively, to obtain query result, and returns to the conjunction
And node；
The merge node merges the query result, to obtain final query result.
8. a kind of instant disposal system for treating, which is characterized in that including merge node and at least one local node；Wherein,
The merge node, for obtaining the real time data of real time processing system transmission；And according to distribution policy, determine one
Handle node；Wherein, the processing node includes a local in the merge node or at least one described local node
Node；
The processing node for creating the full-text index file of the real time data, and monitors the state of the real time data
Condition；If the status condition meets the second Writing condition, the instant disposal system for treating is written into the full-text index file
In slow storage device.
9. instant disposal system for treating according to claim 8, which is characterized in that the merge node and at least one described sheet
Ground node forms distributed cloud framework.
10. instant disposal system for treating according to claim 8, which is characterized in that the processing node is also used to
If the status condition meets the first Writing condition, the fast of the instant disposal system for treating is written into the full-text index file
In speed storage equipment.
11. instant disposal system for treating according to claim 10, which is characterized in that the processing node is specifically used for
If the receiving time of the real time data reaches the first maximum SEE time, alternatively, the quantity of the real time data reaches
First maximum number of files, the processing node set the quick storage that the instant disposal system for treating is written in the full-text index file
In standby.
12. instant disposal system for treating according to claim 8, which is characterized in that the processing node is specifically used for
If the receiving time of the real time data reaches the second maximum SEE time, alternatively, the quantity of the real time data reaches
Second maximum number of files, the processing node set the storage at a slow speed that the instant disposal system for treating is written in the full-text index file
It is standby.
13. instant disposal system for treating according to claim 10, which is characterized in that the processing node is also used to
If the status condition meets first Writing condition, start new query engine, to carry out the real time data
Immediate inquiring.
14. instant disposal system for treating according to claim 8, which is characterized in that the processing node is also used to
If the status condition meets second Writing condition, start new query engine, to carry out the real time data
Immediate inquiring.
15. instant disposal system for treating described in 3 or 14 according to claim 1, which is characterized in that the merge node is also used to
Data inquiry request is received, includes querying condition in the data inquiry request；
Data inquiry request is distributed at least one described local node, so that each at least one described local node
Each local node and the merge node in local node or at least one described local node execute inquiry item respectively
Calculating operation corresponding to part, to obtain query result；
Obtain the query result；
The query result is merged, to obtain final query result.
16. a kind of processing system of real time data, which is characterized in that any including real time processing system and claim 8~14
Instant disposal system for treating described in claim；Wherein,
The real time processing system, for sending the real time data to the instant disposal system for treating.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410229319.1A CN105335362B (en) | 2014-05-28 | 2014-05-28 | The processing method and system of real time data, instant disposal system for treating |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410229319.1A CN105335362B (en) | 2014-05-28 | 2014-05-28 | The processing method and system of real time data, instant disposal system for treating |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105335362A CN105335362A (en) | 2016-02-17 |
| CN105335362B true CN105335362B (en) | 2019-06-11 |
Family
ID=55285906
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410229319.1A Active CN105335362B (en) | 2014-05-28 | 2014-05-28 | The processing method and system of real time data, instant disposal system for treating |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105335362B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10122788B2 (en) * | 2016-03-29 | 2018-11-06 | Amazon Technologies, Inc. | Managed function execution for processing data streams in real time |
| CN107391541B (en) * | 2017-05-16 | 2020-10-20 | 创新先进技术有限公司 | Real-time data merging method and device |
| CN111310170A (en) * | 2020-01-16 | 2020-06-19 | 深信服科技股份有限公司 | Anti-leakage method and device for application program and computer readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102118405A (en) * | 2009-12-31 | 2011-07-06 | 比亚迪股份有限公司 | P2P (Peer-to-Peer) network system applied to real-time video data transmission |
| CN102880475A (en) * | 2012-10-23 | 2013-01-16 | 上海普元信息技术股份有限公司 | Real-time event handling system and method based on cloud computing in computer software system |
| CN103152287A (en) * | 2013-03-27 | 2013-06-12 | 恒生电子股份有限公司 | Method and device for reliably receiving real-time data |
| CN103338261A (en) * | 2013-07-04 | 2013-10-02 | 北京泰乐德信息技术有限公司 | Storage and processing method and system of rail transit monitoring data |
| CN103560943A (en) * | 2013-10-31 | 2014-02-05 | 北京邮电大学 | Network analytic system and method supporting real-time mass data processing |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7613848B2 (en) * | 2006-06-13 | 2009-11-03 | International Business Machines Corporation | Dynamic stabilization for a stream processing system |
-
2014
- 2014-05-28 CN CN201410229319.1A patent/CN105335362B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102118405A (en) * | 2009-12-31 | 2011-07-06 | 比亚迪股份有限公司 | P2P (Peer-to-Peer) network system applied to real-time video data transmission |
| CN102880475A (en) * | 2012-10-23 | 2013-01-16 | 上海普元信息技术股份有限公司 | Real-time event handling system and method based on cloud computing in computer software system |
| CN103152287A (en) * | 2013-03-27 | 2013-06-12 | 恒生电子股份有限公司 | Method and device for reliably receiving real-time data |
| CN103338261A (en) * | 2013-07-04 | 2013-10-02 | 北京泰乐德信息技术有限公司 | Storage and processing method and system of rail transit monitoring data |
| CN103560943A (en) * | 2013-10-31 | 2014-02-05 | 北京邮电大学 | Network analytic system and method supporting real-time mass data processing |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105335362A (en) | 2016-02-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105915650B (en) | Load-balancing method and device | |
| CN107291928A (en) | A kind of daily record storage system and method | |
| CN105491078B (en) | Data processing method and device, SOA system in SOA system | |
| CN108123866B (en) | Method for message transmission and device | |
| CN110289999B (en) | Data processing method, system and device | |
| CN109600798A (en) | Multi-domain resource allocation method and device in a kind of network slice | |
| CN104468838B (en) | A kind of P2P networkings and the web service selection method of publish-subscribe model | |
| de Araujo et al. | VCube-PS: A causal broadcast topic-based publish/subscribe system | |
| CN106161643B (en) | Task distribution formula processing method, device and Cloud Server | |
| EP2958277A1 (en) | Instant messaging method and system | |
| CN110557416B (en) | Multi-node collaborative blocking method and system | |
| CN114024972B (en) | Long connection communication method, system, device, equipment and storage medium | |
| CN109788020A (en) | One kind is attended a banquet distribution method and relevant device | |
| CN104424240B (en) | Multilist correlating method, main service node, calculate node and system | |
| CN109450987A (en) | Number generation method, apparatus and system and storage medium | |
| CN105335362B (en) | The processing method and system of real time data, instant disposal system for treating | |
| CN104202386B (en) | A kind of high concurrent amount distributed file system and its secondary load equalization methods | |
| CN104426968B (en) | Data managing method and device | |
| CN109104406A (en) | A kind of online customer service management method, device, server and storage medium | |
| CN105978938A (en) | Service processing equipment service status determining method and scheduling equipment | |
| CN108563697A (en) | A kind of data processing method, device and storage medium | |
| CN106899621B (en) | A kind of scheduling system and method | |
| CN104301241B (en) | A kind of SOA dynamic load distributing methods and system | |
| CN111953551A (en) | Log data transmission method and device, electronic equipment and storage medium | |
| CN106874371A (en) | A kind of data processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20211125 Address after: No. 699, Wangshang Road, Binjiang District, Hangzhou, Zhejiang Patentee after: Alibaba (China) Network Technology Co.,Ltd. Address before: Box 847, four, Grand Cayman capital, Cayman Islands, UK Patentee before: ALIBABA GROUP HOLDING Ltd. |