CN104838377A - Integrating event processing with map-reduce - Google Patents

Abstract

Large quantities of data can be processed and/or queried relatively quickly using a combination of continuous event processing and a Map-Reduce algorithmic tool. The continuous event processor can continuously produce real-time results by merging (a) CQL query results from events received since a currently executing Map-Reduce job was started with (b) a most recent query result produced by a most recently completed Map-Reduce job. When the currently executing Map-Reduce job completes, its query result can be stored and made accessible to the continuous event processor, and a new Map-Reduce job can be started relative to event data that has grown in size since the execution of the last Map-Reduce job. The Map-Reduce algorithmic tool provides a convenient mechanism for analyzing and processing large quantities of data.

Description

Utilize and map the process of reduction integration events

To the cross reference of related application

According to 35U.S.C.119 (e), the application requires that the title submitted on Dec 5th, 2012 is the U.S. Provisional Application No.61/733 of " A METHOD FOR INTEGRATING EVENT PROCESSINGWITH MAP REDUCE ", the right of priority of 844, its full content is incorporated into this by reference, for any object.According to 35U.S.C.119 (e), the application also requires that the title submitted on March 29th, 2013 is the U.S. Provisional Application No.61/806 of " INTEGRATING EVENT PROCESSING WITH MAP REDUCE ", the right of priority of 744, its full content is incorporated into this by reference, for any object.According to 35U.S.C.119 (e), the application also requires that the title submitted on November 13rd, 2013 is the U.S. non-provisional application No.14/079 of " INTEGRATING EVENTPROCESSING WITH MAP-REDUCE ", the right of priority of 538, its full content is incorporated into this by reference, for any object.

Background technology

Continuous events processor can receive continuous print flow of event and process each event by inquiring about each event application continuous events process (CEP) wherein comprised.This CEP inquiry can be formatted as the grammer meeting CEP query language, and wherein CEP query language such as continuous-query language (CQL), it is the expansion of Structured Query Language (SQL) (SQL).SQL query is applied once the data be stored in the table of relevant database (each user's request) usually, and CQL inquires about along with the event in arrival flow of event is repeatedly applied to those events by the reception of continuous events processor.

Such as, data stream can specify the stock price of various company.As time goes by, the up-to-date stock price of those companies can be added in data stream.Continuous events processor can receive each this stock price, and inquires about that stock price application CQL when that stock price arrives.CQL inquiry can specify various operation, comprises filtration and converging operation potentially.Then continuous events processor can export the application result of CQL inquiry to various interested listener.

Accompanying drawing explanation

Fig. 1 illustrates for utilizing the combination mapping Reduction algorithm instrument and CEP to calculate process flow diagram relative to the technology of the real-time query result of mass data according to embodiments of the invention.

Fig. 2 illustrates for utilizing the combination mapping Reduction algorithm instrument and CEP to calculate block diagram relative to the system of the real-time query result of mass data according to embodiments of the invention.

Fig. 3 is the simplified block diagram of the assembly that the system environments that can use according to embodiments of the invention is described.

Fig. 4 is the simplified block diagram of the computer system that can use according to embodiments of the invention.

Fig. 5 illustrates for generating continuously and the process flow diagram of the example of the technology of batch processing assembly based on single query according to embodiments of the invention.

Fig. 6 illustrates for the process flow diagram of example based on the query generation functionally technology of the operational symbol of equivalence according to embodiments of the invention.

Fig. 7 is the simplified block diagram of the electronic equipment 700 according to the embodiment of the present invention.

Embodiment

Embodiments of the invention relate to continuous events process field.Embodiments of the invention comprise the technology for utilizing continuous events process rapidly process relative to the combination mapping reduction (Map-Reduce) algorithmic tool or inquiry mass data.In an embodiment, continuous events processor reduces by the mapping completed recently the nearest Query Result that operation produces from the CQL Query Result of event received after the mapping reduction job initiation of current execution and (b) produce real-time results continuously by merging (a).When the mapping reduction operation of current execution completes, store its Query Result and make this Query Result be addressable for continuous events processor, and starting new mapping reduction operation relative to the event data that size after upper maps reduction Job execution has increased.Mapping Reduction algorithm instrument is that treatment and analysis mass data provides mechanism easily.This large amount of data are sometimes referred to as " large data ".

Large data are the terms used by infotech media and supplier.Enterprise is accumulating Shuo Ju – – billion (terabytes) byte of enormous amount, even petabyte (petabytes) Shuo Ju – – and there is competition desire obtain more understanding in depth market trend and company performance.Company is faced with management and analyzes the hope of large data.The data of modern application process huge amount.These huge amounts make to be difficult to utilize conventional treatment mechanism promptly to process data.

As the example of large data processing, can consider that wherein sales manager wants the situation of the most best-selling product of identification company each principal market that product can be purchased in the time period (such as last month) extended wherein.The challenge completing this task is will data volume handled by this analysis.This data volume may thousands of, 1,000,000, between even billions of sales transactions not etc.The general purpose tool of such as relevant database and visual software can not be stretched over so a large amount of data usually.In addition, data can be structurized and non-structured, and this may not be the feature handled by data management and analysis tool.Conventional system R can not process so a large amount of data in time.

Therefore, in order to manage large data, can be carried out some and attempt.The method of a large amount of isomeric data of potential process divides across different servers and copies data, to create more manageable data set.But the multiple copies creating data make the consistance being more difficult to maintenance information, because may can not copy other data set to the renewal of a data set.

This consistance shortage can cause, and such as, if the application of usage data upgrades the entry in sales data, but then that renewal fails to copy another website copied to.If other application of accessing these data employs some these data in other transaction, so situation can become complicated.When data change, repartitioning of data may be suitable, but wherein this to repartition the mode that be performed normally unclear.

It is shielded, unmodifiable data that a kind of possible method for the treatment of these complicacy can comprise Data distribution8 (distributing).Such as, sales transactions can be stored in certain sustainable memory storage as they occur.Can prevent from applying the raw data changing that and store.Make data keep immutable and avoid consistency problem, and distribution relaxes the problem of amount.But, make the immutable meeting of data make existing data processing tools can not with expect mode process distributed (distributed) data.

Specific distributed algorithm instrument with unmodifiable distributed input in conjunction with work, and can be similar to relative to distributed system execution the inquiry described in above sale example.This specific algorithmic tool maps Reduction algorithm instrument.Exist and map the various different implementation of Reduction algorithm instrument.Such as, a kind of implementation can find in Open Source Code ApacheHadoop.Utilize Apache Hadoop, can create and use the transaction of large data selling as input and then return most best-selling product mapping reduction operation as a result.

Map Reduction algorithm instrument relative to distributed data work.Data can be divided (partition) in independent node, and each node can be maintained as independent file on independent computing machine.Each this machine can perform inquiry abreast relative to the data safeguarded on that machine and other machine.Therefore, each machine can produce the independent Query Result about its particular data divides.Then, these independent results can be reduced into single unified result.Map Reduction algorithm instrument and can complete this reduction.For each entry in data, mapping Reduction algorithm instrument can by that entry map to result.Such as, if Data Entry represents product, then map Reduction algorithm instrument and each product can be mapped to the result of that product of instruction relative to the sale rank of other products.Then, map Reduction algorithm instrument and all these results can be merged into single unified result.This merging is called as reduction (reduction).

How relative to the example of data manipulation, can consider to relate to the example of the appearance quantity calculating certain words in webpage independent in a large number for mapping Reduction algorithm instrument.First webpage may comprise 10 quoting certain words.Second webpage also may comprise 10 quoting certain words.3rd webpage may only include single quoting certain words.4th webpage may comprise two quoting certain words.Each webpage can be mapped to the appearance quantity of the certain words in that webpage by the mapping process mapping the execution of Reduction algorithm instrument.In this example, only comprised all quantity being mapped to each webpage being sued for peace together by the reduction process mapping the execution of Reduction algorithm instrument, to produce final result, in this example, result is that 23 times of certain words occur.

Owing to mapping the distributed nature of the data that Reduction algorithm instrument operates thereon, and depend on the size of input data, map the every n minute of reduction operation possibility (with reference to sale example discussed above), or, more possibly, within every n hour, return with batch result the item that is in great demand most.In some cases, several minutes or a few hours are oversize and can not wait for for Query Result.Such as, the application of some fraud detection requires in the result immediately when carrying out of concluding the business, instead of after thousands of fraudulent trading has completed.Such as another example, retailer may wish to have the up-to-date understanding to the poorest sell goods, so that those products of sales promotion can be carried out by providing promotional card to when client enters shop those clients in shop.Along with waiting for that the time quantum that knowledge spends increases, the possible irrelevance of that knowledge also increases.The implementation of typical mapping Reduction algorithm does not produce real-time query result.

According to embodiments of the invention, continuous events process (CEP) technology can be combined with mapping Reduction algorithm instrument, to be that the inquiry performed for large-scale distributed data set produces Query Result faster.Utilize CEP, continuous events processor receives event in real time from data stream, and along with they are received, relative to those event execution analysis process.Such as, along with sales transactions occurs in the shop of company, each transaction in those transaction can be added in data stream as event in the moment of those transaction.The event of placing in a stream can be monitored by the reception interested entity of those events (such as application program).These entities can to the registration of continuous events processor to receive this event.In addition, along with this sales transactions occurs in the shop of company, each transaction in those transaction can be added in the persistent data memory block of such as relevant database, and it can distribute according to certain partition mode across multiple independently website.

Along with those events arrive continuous events processor, CEP can be inquired about the event be applied to from flow of event by this continuous events processor continuously.Such as, processor can be determined to the event application representing sales transactions in the inquiry with the most best-selling product on the time window of specifying the duration.In order to do like this, processor can be safeguarded about processor in the data with all events received from data stream in the time window of the nearest generation of that duration in computing store.But, the length of this time window can by continuous events processor can the amount of computing store limit.Usually, continuous events processor comprises all sales transactions once occurred so that time window is extended to, such as, if especially there occurs very a large amount of sales transactions by not having enough storeies.Equally, the amount of depending on, continuous events processor even may not have enough storeies to comprise all sales transactions occurred in last year so that time window is extended to.In addition, along with the increase of time window size, continuous events processor performs inquiry institute's time spent amount relative to all events fallen in this window also to be increased.CEP mechanism is used to calculate the result about event in the time window with the relatively limited duration traditionally, is only the magnitude of a few hours or several minutes sometimes.

In order to overcome restrictions more discussed above, the ability of CEP and the ability mapping Reduction algorithm instrument combine by embodiments of the invention.What map that Reduction algorithm instrument may be used for accumulating recently for event data batch calculates Query Result.Such as, can dispatch and map reduction operation and calculate Query Result relative to the event data stored in distributed relation database; This data by persistent, mode in nonvolatile manner, can store usually on the magnetic medium of such as hard disk drive.This operation may spend the relatively long time to complete.Such as, depend on the amount of the event data of accumulating in Distributed Storage device, operation may spend over one day and complete.In an embodiment of the present invention, start to perform in the time of plan once map reduction operation, map the event data being not later than planned time that Reduction algorithm instrument is just only considered to have stored in Distributed Storage device; Mapping reduction operation does not comprise the event when mapping Reduction algorithm instrument at processing operation in current arrival data stream; According to embodiment, map reduction operation comprise by this mapping reduction operation start to be performed time received from data stream and be stored in the event in Distributed Storage device.

Each mapping reduces the example that operation is batch technology.Assuming that the time mapping reduction operation cost specified amount completes, so to arrive in data stream and the event be stored in Distributed Storage device takes at the mapping reduction Zuo Ye Zhong – – performed subsequently in next batch at the time durations of specified amount.According to embodiments of the invention, after continuous events processor has started to be processed relative to the mapping reduction in (and, in one embodiment, only exist) current execution already, the event arrived in data stream performed CEP continuously.In such an embodiment, continuous events processor performs the same or similar inquiry of inquiry performed as a part of planning operation relative to the event stored in Distributed Storage device before with mapping Reduction algorithm instrument relative to those newly arrived events.Along with those events arrive data stream, continuous events processor can perform this Cha Xun – – relative to newly arrived event thus, CEP inquiry can perform in real time.In an embodiment of the present invention, when each continuous events processor produces Query Result, that Query Result and the mapping completed recently are reduced the nearest result that operation produces and merge, to produce unified Query Result by processor.Unified Query Result reflects relative to the event data be stored in Distributed Storage device and the query processing performed than the event data that memory storage more closely arrives.

Fig. 1 illustrates for utilizing the combination mapping Reduction algorithm instrument and CEP to calculate process flow diagram relative to the technology 100 of the real-time query result of mass data according to embodiments of the invention.Although technology 100 is shown as including some operation performed with particular order, alternative of the present invention can comprise that add with different order execution potentially, less or different operation.

At square frame 102, map Reduction algorithm instrument (such as, Apache Hadoop) and start to perform mapping reduction operation relative to the event data be stored in Distributed Storage device.In an embodiment, map reduction operation to comprise and perform the inquiry of specifying relative to the specified subset of all events be stored in Distributed Storage device.This subset can be included in the set of the whole event stored in Distributed Storage device, comprising, mapping reduction operation (if any) since before starts to perform the event be later stored in Distributed Storage device.

At square frame 104, when mapping Reduction algorithm instrument and performing mapping reduction operation relative to the event data be stored in Distributed Storage device, continuous events processor performs CQL inquiry continuously for the event arrived continuously through data stream.It is all events received in the fixed time window of end point that CQL inquiry can act on having with current time.CQL inquires about the data can be polymerized in its execution from these events.In an embodiment, CQL inquiry is identical or similar with the inquiry being reduced Job execution by the mapping of current execution or perform identical function, but relative to different events.In an embodiment, all events (it is by the mapping of current execution reduction operation process) be stored in Distributed Storage device are got rid of in CQL inquiry from its process.As discussed above, in an embodiment, these events are also stored in Distributed Storage device, but do not consider in the execution of the mapping reduction operation of current execution.

At square frame 106, continuous events processor calculates preliminary Query Result based on nearest execution of the CQL inquiry relative to the data event at the appointed time window.

At square frame 108, if any mapping reduction operation completes, then the result that preliminary Query Result and the mapping completed recently reduce operation merges by continuous events processor.This merging produces unified Query Result.In an embodiment of the present invention, complete if also do not map reduction operation, then preliminary Query Result becomes unified Query Result.

At square frame 110, unified Query Result is outputted to interested monitoring entity by continuous events processor.Such as, unified Query Result can output to and register with the application program receiving this unified Query Result to continuous events processor by continuous events processor.

At square frame 112, make the decision whether completed about the mapping reduction operation started recently.If the mapping reduction started recently completed already, then controlled to be delivered to square frame 114.Otherwise, control to turn back to square frame 104.

At square frame 114, the result mapping reduction operation is preserved enduringly.Such as, the result mapping reduction operation can be stored on the hard disk drive that continuous events processor can access.As above in conjunction with square frame 108 discuss, the PRELIMINARY RESULTS of this result and processor oneself can be merged to produce unified Query Result by continuous events processor.In an embodiment, the result of mapping reduction operation covers and replaces it the result stored before of the front mapping reduction operation performed.

At square frame 116, map Reduction algorithm instrument (such as, Apache Hadoop) and start to perform new mapping reduction operation relative to the event data be stored in Distributed Storage device.In an embodiment, map reduction operation to comprise and perform the inquiry of specifying relative to the specified subset of all events be stored in Distributed Storage device.In an embodiment, this subset comprises since mapping reduction operation before starts to perform the event be stored in Distributed Storage device later.Control to transmit back side frame 104.

Such as, if the quantity of result instruction certain words in a web pages of the mapping reduction operation completed recently is 23, and if new webpage arrives in data stream as event, and if this certain words determined by continuous events processor, the quantity in that new web page is 5, then continuous events processor can merge this reduces job result result from the mapping completed recently, to produce unified Query Result 28.New web page can be added to Distributed Storage device, so that follow-up (not being current) mapping reduction operation can operate in the event data comprising this new web page.

Fig. 2 illustrates for utilizing the combination mapping Reduction algorithm instrument and CEP to calculate block diagram relative to the system 200 of the real-time query result of mass data according to embodiments of the invention.In an embodiment, system 200 comprises event source 202, comprises the data stream 204 of event, continuous events processor 206, maps Reduction algorithm instrument 208, Distributed Storage device 210 and monitor entity 212.In an embodiment, along with those events occur, event source 202 can add event (such as, sales transactions, stock price etc.) to data stream 204 continuously.Continuous events processor 206 can read event continuously from data stream 204, and can perform CQL inquiry relative to those events continuously.Continuous events processor 206 can also store it continuously and read event in Distributed Storage device 210.The execution result that continuous events processor 206 can also merge CQL inquiry continuously and the mapping completed performed by mapping Reduction algorithm instrument 208 reduce the latest result of operation.Map Reduction algorithm instrument 208 and can perform mapping reduction operation one by one.Each this mapping reduction operation can perform relative to mapping by that event be stored in Distributed Storage device 210 when reduction operation starts to perform and CQL inquire about same or analogous inquiry.When each this mapping reduction operation completes, it can be that addressable medium (potential Distributed Storage device 210) goes up the result of preserving its query execution, for the object merged concerning continuous events processor 206 enduringly.Monitor entity 212 and can receive the unified Query Result exported by continuous events processor 206 continuously, and use those to show this result to human user potentially.

In one embodiment, CQL inquiry and corresponding mapping reduction (such as, Hadoop) operation can be created individually and manually by user and perform.But in alternative embodiment, establishment and the execution of CQL inquiry and corresponding mapping reduction operation can be integrated, make it possible to automatically generate to map based on the CQL inquiry of existing correspondence reduce operation (or program).Such as, in an embodiment, in order to create the CQL inquiry determining most best-selling product based on the sales transactions event represented in a stream, user can utilize CEP system.Then map reduction program can utilize CQL to inquire about as inputting automatic generation by mapped reduction program generator.Program (or identical program or independent program) also can be generated automatically, and reduces the result that operation returns and the result that the execution inquired about by CQL returns to merge continuously by the mapping of each execution.

Mapping reduction operation can with being called that the language-specific of Pig Latin represents.PigLatin is high level descriptive language.Such as, in Pig Latin, operation or program can be expressed and load data, filter those data, by those data and other data cube computation etc.Pig Latin program can be expressed according to operational symbol.These operational symbols can perform the operation such as loading data, filtering data, connection data etc.In an embodiment of the present invention, Pig Latin program generator receives CQL inquiry as input, and is automatically in its assembly operational symbol by that query decomposition.This CQL operational symbol can comprise filtration operation symbol, concatenation operator, project symbol etc.In an embodiment of the present invention, Pig Latin program generator utilizes the mapping between CQL operational symbol and Pig Latin operational symbol, automatically one group of operational symbol is transformed into another group operational symbol; This mapping can comprise the mapping of man-to-man operational symbol to operational symbol, and the operational symbol of one-to-many to the mapping of operational symbol and many-to-one operational symbol to the operational symbol of the mapping of operational symbol and multi-to-multi to the mapping of operational symbol.

In an embodiment, after the CQL operational symbol based on correspondence automatically generates one group of PigLatin operational symbol, Pig Latin program generator automatically generates and stores the stream that can comprise CQL operational symbol and Pig Latin operational symbol.This stream can indicate the output of CQL operational symbol will become input to Pig Latin operational symbol, and/or the output of Pig Latin operational symbol will become the input to CQL operational symbol.Therefore, can generate the digraph of operational symbol, this digraph comprises Pig Latin and CQL operational symbol.In this digraph, the limit between operational symbol node can the flowing of designation data from operational symbol to operational symbol.Potentially, an operational symbol can export data to multiple operational symbol, and multiple operational symbol can export data to single operational symbol.Leaf node operational symbol in figure can perform CQL operating result and map the final merging between reduction operating result.After automatically generating digraph by this way, automatically can perform some optimization of figure, comprise some the operational symbol nodes in the operational symbol type merging figure performed based on those operational symbols potentially.Such as, single operational symbol is become together with the CQL operational symbol all performing filter operation can be incorporated in Pig Latin operational symbol.

When performing, can make an explanation to digraph.When the one group of Pig Latin operational symbol running into Pig Latin operational symbol in the drawings or connect by one or more side chain, the mapping reduction program performing the correspondence of the operation of those operational symbols can be performed automatically.The CQL that the result mapping reduction program can be sent to corresponding to CQL operational symbol in digraph as incoming event inquires about.

Single language is utilized to connect continuously and the result of batch processing

As discussed above, in one embodiment, continuous query processing device can process continuously for the event in current arrival event stream CQL inquiry, and map before reduction program processes simultaneously from stream receive and the event stored.The result of the process performed by continuous query processing device can merge termly with by the result mapping the process that reduction program performs, to generate the relatively new result to large data collection.

According to embodiment, continuous query processing device performs the inquiry of the single high level language with such as CQL.As the compiling result of this inquiry, two aspects operationally can be generated.An aspect performs event stream processing.At same compile duration, as on the other hand, additional operational symbol can be generated.These additional operational symbols can be integrated into Hadoop or some map in reduction instrument.Map reduction instrument can perform towards batch process.The result of stream process can merge with the result of batch processing.Therefore, a large amount of data can be handled also allow to obtain result in real time simultaneously.

Event handling is applied, and such as Oracle Event Processing (Oracle event handling, OEP), is designed to process event in real time.But this application is designed to the event processing relative brevity existence traditionally.This process is not very deep traditionally.Event handling application processes the event when processing generation traditionally.Those event handlings application does not check the data that may store several years ago in a database traditionally.

Map reduction instrument, such as Hadoop, can the mass data in the storage vault being stored in such as database operate, wherein storage vault can physically be included in hard disk drive.Be stored in data in this database can relatively Chen Jiu – – even the several years old.The data be stored in this database may produce and store in the time before.Because it may have accumulated for a long time, the amount being therefore stored in the data in this database can be relatively large.Map reduction instrument and can process the large data collection stored in a distributed way in a database.Map reduction instrument and can create mapping (map) task and reduction (reduce) task.These tasks can perform in a distributed fashion.The process of large data collection can be broken down into independent aspect, and can be distributed in independent treatment element the process of each aspect.Map reduction instrument can with towards batch mode perform its process, the existing beginning of batch job that it is performed relative to large data collection has end again.

Mapping reduction instrument can perform it in the mode be separated with the process performed by continuous events processor simultaneously and process.Although map reduction instrument with have that clear and definite task starts and terminates towards batch mode perform its process, continuous events processor performs it continuously and processes, and does not terminate.Event in flow of event arrives continuous events processor continuously.Embodiments of the invention can link together the result of the result of the continous way process of continuous events processor with the batch-type process mapping reduction instrument.In addition, embodiments of the invention can utilize the single higher level lanquage of such as CQL to realize this connection.

Alternative method can comprise the application performing stream process and another the independent application performing batch processing.The 3rd customized application can be write attempt dissimilar result to put together.This method may relate to the different volume journey language speech – – mono-kind of use three kinds for flowing process, a kind of for batch processing and a kind of result for connecting every type.

In order to avoid using multiple different language, a kind of embodiment of the present invention comprises the mapping reduction program senior CQL query compiler being become the first-class effect of function.Then the execution result of CQL inquiry merges with the result mapping reduction program.CQL inquiry can bear results in real time, and maps reduction program and can perform more deep analysis on larger data set.Therefore, the result of merging is real-time and deep.This merging can realize based on the inquiry of specifying with the unique user analysing in depth assembly being in real time compiled as generation system.Do not need the independent application that structure different language is write.

Fig. 5 illustrates for generating continuously and the process flow diagram of the example of the technology 500 of batch processing assembly based on single query according to embodiments of the invention.At square frame 502, receive the inquiry of specifying with CQL.At square frame 504, the continuous events processing execution plan that CQL query compiler becomes to be performed by continuous events processor by compiler.At square frame 506, CQL query compiler is become that function is first-class imitates in the plan of continuous events processing execution and can as the batch job mapping tasks that performs of mapped reduction program and reduction task in a distributed fashion by compiler.At square frame 508, compiler generates the merge module of the result that can connect and be produced by continuous events processor and the result produced by mapping reduction program.At square frame 510, continuous events processor relative to the current arrival event executive plan from flow of event, and maps reduction program relative to the data execution mapping tasks stored before in a database and reduction task.At square frame 512, merge module connects by continuous events processor and the result mapping the generation of reduction program.Control to transmit back side frame 510.

Generate in real time corresponding and batch operational symbol

In sample scenario, user may want to count the quantity of quoting of certain words in one group of text.Map reduction instrument, such as Hadoop, can process each file in one group of file, to count the quantity of quoting of certain words.This group file of mapped reduction instrument process may comprise the short text stored from flow of event several years ago.Such as, outside relatively near past (such as, nearest minute), the short text that the span that this group file may be included in 10 years receives from stream.When this happens, continuous events processor can receive similar short real-time text from the application on site of such as Facebook or Twitter.Continuous events processor can after those real-time texts are generated soon (such as, after user utilizes Facebook or Twitter to put up these texts immediately) to these real-time texts in the quantity of quoting of certain words is counted.Can be connected with the counting calculated by continuous events processor by the counting mapping the calculating of reduction instrument.

In order to specify the word count task that will perform, user can use CQL to carry out given query.Grammatically, CQL is similar to Structured Query Language (SQL) (SQL).Inquiry can generally adopt following form, such as: FROM location GROUP BY wordSELECT count.This inquiry can collect all sentences from the position of specifying, count the quantity of the word in every group by assigning to from the unique word these sentences in different groups, then.Therefore, inquiry can utilize relatively simple form to specify.

Query decomposition can be become independent operational symbol by the compiler receiving the CQL inquiry that user specifies.These operational symbols can comprise, such as, polymerization (aggregation) operational symbol that grouping (group-by) operational symbol, execution count, output (output) operational symbol of also serving as source (source) operational symbol, etc.Based on this group operational symbol, compiler can generate (a) query execution plan for real-time event process, and (b) is for the treatment of the batch job of large data collection.With reference to example word count sight concrete above, the generation of batch job can comprise the generation various sentence being mapped to the map operator in independent unique word.The generation of batch job can also comprise will often organize unique word and reduce into the generation of the reduction operational symbol of the respective digital of those word quantity of expression.As a part for generated query executive plan, compiler can generate merging (merge) the operational symbol batch reduction result of operational symbol and the result of continuous polymerization operational symbol merged.Then batch job can be inquired about with real-time continuous and perform simultaneously.Union operation symbol obtains batch view and the RUNTIME VIEW from continuous events processor from mapping reduction assembly.

Fig. 6 illustrates for the process flow diagram of example based on the query generation functionally technology 600 of the operational symbol of equivalence according to embodiments of the invention.At square frame 602, the currentElement of inquiry is set to first element inquired about.At square frame 604, determine the type (such as, grouping, polymerization, selection etc.) of currentElement.At square frame 606, based on the type selecting continuous events process operational symbol type of currentElement.At square frame 608, based on the type selecting batch processing operational symbol type (such as, mapping, reduction etc.) of currentElement.At square frame 610, the operational symbol of the continuous events process operational symbol type selected by generation, and added to continuous events processing plan.At square frame 612, the operational symbol of the batch processing operational symbol type selected by generation, and added to batch processing plan.At square frame 614, after being made at currentElement, whether this inquiry comprises the decision of any element.If so, then control to be delivered to square frame 616.Otherwise, control to be delivered to square frame 618.

At square frame 616, the currentElement of inquiry is set to the next element inquired about.Control transmission and get back to square frame 604.

Alternately, at square frame 618, generate union operation symbol, and added to continuous events processing plan.Then, the technology shown in Fig. 6 completes.

Stateless and have state operational symbol

Usually, in CQL, there are two groups of different operational symbols.One group of this operational symbol is stateless.Utilize this stateless operational symbol, an input is mapped to the one or more outputs in the figure of the operational symbol of interconnection usually.According to embodiments of the invention, stateless CQL operational symbol corresponds to the mapping operations symbol in batch processing field.Another group CQL operational symbol has state.Utilize this stateless operational symbol, multiple input is mapped to an output in the figure of the operational symbol of interconnection usually.According to embodiments of the invention, state CQL operational symbol is had to correspond to the reduction operational symbol in batch processing field.

Therefore, in an embodiment of the present invention, compiler determines whether CQL searching elements is mapped to one or more output by an input.If so, then compiler determines that this element corresponds to stateless event handling operational symbol.Filtration operation symbol is the example of stateless event handling operational symbol.In response to determining that event handling operational symbol is stateless operational symbol, compiler generates the mapping operations symbol (the mapping reduction language with selecting) for being included in the functionally equivalence in batch processing job.Alternately, compiler can determine whether CQL searching elements is mapped to single output by multiple input.If so, then compiler determines that this element is corresponding to the event handling operational symbol having state.Window (window) operational symbol is the example of the event handling operational symbol having state.In response to determining that event handling operational symbol is the operational symbol having state, compiler generates the reduction operational symbol (the mapping reduction language with selecting) for being included in the functionally equivalence in batch processing job.

The element of CQL inquiry can perform the polymerizable functional (such as, summation (sum), average (average), counting (count), etc.) of particular type.In an embodiment of the present invention, compiler determines that CQL inquires about the specific aggregation type of aminated polyepichlorohydrin symbol, and responsively generates the reduction operational symbol performing identical type polymerizable functional.Therefore, such as, if continuous events operational symbol performs sum operation, then compiler responsively generates the reduction operational symbol performing sum operation, if but continuous events operational symbol performs counting operation, then compiler responsively generates the polymerization operator performing counting operation.

Avoid the repeated events process in amalgamation result

According to embodiments of the invention, to make the patten's design union operation symbol do not overlapped between real-time continuous result and non real-time batch processing result.Union operation symbol can comprise safeguard measure, and it prevents any data item due to by real time and non real-time two aspect process and merging with self.Therefore, in any particular iteration process of the result connected, individual event will not be had and be considered more than once.In an embodiment of the present invention, in any specific iterative process, union operation symbol only accepts from than the time point of specifying (such as, before one minute) old non real-time (such as, map reduction) result of assembly, and the result only accepted from real-time (such as, continuous events processor) assembly old unlike the time point of specifying.Because process can be repeatedly performed, therefore time point can advance along with each iteration.

Such as, in the first iteration, the border that union operation symbol can define large data collection only is included on current date until 12:50:45 event.Union operation symbol can define first time only include event nearer than the 12:50:45 on current date by the border of the data set of continuous events processor process in this iteration.Assuming that be ten seconds for the time window of continuous events processor, then for second time iteration, this border can advance ten seconds kinds to 12:50:55 from 12:50:45.The event that (across in ten second time interval of 12:50:45 to 12:50:55) receives in first time iterative process has been added to large data collection when they arrive, and has been labeled timestamp to indicate it from the time of arrival of flow of event.In the end of each iteration, border can advance the time quantum needed for real-time event process of a unit.This amount can based on to continuous events processor can the size of storer.In an embodiment of the present invention, this time quantum is not static, but can iterate to the change of another iteration from one, makes border can advance different amounts after different iteration.In such an embodiment, the border of each iteration can based on will by the event number of specifying of continuous events processor process, instead of the duration on regular time.

Therefore, in any specific iterative process, union operation symbol is guaranteed can not be comprised by the event sets that continuous events processor processes in that particular iteration process any event being also batch processing device and processing in the specific iterative process that is identical, although these events can be included in afterwards by the event sets that processed in iterative process subsequently by batch processor.In essence, union operation symbol can calculate the associating (union) of the results set produced by continuous events processor and batch processor, and just in unified set, the timestamp of event does not allow overlap.

It should be pointed out that union operation symbol may not only only do in some cases and link (concatenate) or sue for peace real-time results and batch result.Such as, if comprise average calculating operation symbol in processes, then union operation symbol can perform more complicated merging.Such as example more specifically, if continuous events processor determines that four digital mean values are 4, and if batch processor determines that eight digital mean values are 10, then union operation symbol will determine that the mean value of all ten two-digits is (4*4+8*10)/12=8.This merging performed can depend on the type of the converging operation be performed in queries.

CQL query compiler is become Pig Latin program

Pig Latin mappedly can reduce the language used.In an embodiment of the present invention, the compiling of CQL inquiry can be included in Pig Latin and generate simulation trial symbol.Accord with based on these simulation trials, the Pig Latin program of functionally equivalence can be compiled device and automatically generate, and compiling is simultaneously used for being inquired about by the CQL that continuous query processing device uses.PigLatin program is performed while the execution that mapping reduction instrument can be inquired about at the CQL of continuous query processing device.

But, in alternative of the present invention, do not generate Pig Latin operational symbol.On the contrary, can generate can this locality (native) operational symbol that uses of mapped reduction instrument.Embodiments of the invention are not limited to the embodiment wherein generating Pig Latin program.

Ardware overview

Fig. 3 is the simplified block diagram of the assembly that the system environments 300 that can use according to the embodiment of the present invention is described.As shown, system environments 300 comprises one or more client computing device 302,304,306,308, and they are configured to operated client application, such as web browser, proprietary client (such as, Oracle Forms), etc.In various embodiments, client computing device 302,304,306 and 308 can be mutual with server 312.

Client computing device 302, 304, 306, 308 can be general purpose personal computer (as an example, comprise and run the Microsoft Windows of various version and/or the personal computer of AppleMacintosh operating system and/or laptop computer), cell phone or PDA (run the software of such as Microsoft Windows Mobile, and enable the Internet, Email, SMS, Blackberry or other communication protocol), and/or run the workstation computer of any various commercial available UNIX or class-UNIX operating system (including but not limited to various GNU/Linux operating system).As an alternative, client computing device 302,304,306 and 308 can be can through network (such as, network 310 described below) carry out other electronic equipment any of communicating, such as thin-client computing machine, enable the games system of the Internet and/or personal messages transfer equipment.Although exemplary system environment 300 is shown to have four client computing device, any amount of client computing device can be supported.Miscellaneous equipment, the equipment etc. of such as belt sensor, can be mutual with server 312.

System environments 300 can comprise network 310.Network 310 can be the network of any type that those skilled in the art are familiar with, and these networks can utilize any various commercial available agreement supported data communication, include but not limited to TCP/IP, SNA, IPX, AppleTalk etc.As just example, network 310 can be LAN (Local Area Network) (LAN), such as ethernet network, token-ring network etc.; Wide area network; Virtual network, includes but not limited to Virtual Private Network (VPN); The Internet; Intranet; Extranet; Public switch telephone network (PSTN); Infrared network; Wireless network (such as, according to any one in the network of IEEE802.11 protocol suite, Bluetooth protocol as known in the art and/or other radio protocol operations any); And/or the combination in any of these and/or other network.

System environments 300 also comprises one or more server computer 312, its can be multi-purpose computer, dedicated server computer (as an example, comprise PC server, unix server, middle-grade server, mainframe computer, rack-mount server etc.), server zone, cluster of servers, or any other suitable layout and/or combination.In various embodiments, server 312 can be adapted to operate in the one or more service or software application that describe in above disclosure.Such as, server 312 can correspond to the continuous events processor 206 of Fig. 2 or map Reduction algorithm instrument 208.

Server 312 can operation system, comprises any operating system discussed above, and any commercial available server OS.Server 312 can also run any various additional server application and/or middle-tier application, comprises http server, ftp server, CGI server, java server, database server etc.Exemplary database server includes but not limited to from commercial available those such as Oracle, Microsoft, Sybase, IBM.

System environments 300 can also comprise one or more database 314,316.Database 314,316 can reside in various different position.As an example, on the one or more storage mediums that can reside in server 312 this locality in database 314,316 (and/or residing in server 312).As an alternative, database 314,316 away from server 312, and can communicate with server 312 through network or special connection.In one group of embodiment, database 314,316 can reside in the storage area network (SAN) that those skilled in the art are familiar with.Similarly, the file for performing any necessity of the function that server 312 has can suitably be stored on server 312 and/or remote storage in this locality.In one group of embodiment, database 314,316 can comprise and is suitable for storing in response to the order of SQL form, upgrades and the relevant database of retrieve data, such as Oracle 10g.

Fig. 4 is the simplified block diagram of the computer system 400 that can use according to embodiments of the invention.Such as, Fig. 2 continuous events processor 206 or map Reduction algorithm instrument 208 system of such as system 400 can be utilized to realize.Computer system 400 is shown as including can through the hardware element of bus 424 electric coupling.Hardware element can comprise one or more CPU (central processing unit) (CPU) 402, one or more input equipment 404 (such as, mouse, keyboard etc.), and one or more output device 406 (such as, display device, printer etc.).Computer system 400 can also comprise one or more memory device 408.As an example, (one or more) memory device 408 can comprise the equipment of such as disk drive, light storage device and the such as solid storage device of random access memory (RAM) and/or ROM (read-only memory) (ROM), they can be programmable, can flash memory upgrade etc.

Computer system 400 can also comprise computer-readable storage media reader 412, communication subsystem 414 (such as, modulator-demodular unit, network interface card (wireless or wired), infrared communication device etc.) and working storage 418, wherein working storage 418 can comprise RAM and ROM equipment as above.In certain embodiments, computer system 400 can also comprise process accelerator module 416, and it can comprise digital signal processor (DSP), application specific processor etc.

Computer-readable storage media reader 412 also can be connected to computer-readable recording medium 410, together (and, alternatively, in conjunction with (one or more) memory device 408) represent long-range, local all sidedly, fixing and/or removable storage device adds storage medium for comprising computer-readable information temporarily and/or more for good and all.Communication system 414 can allow to exchange data with network and/or above-mentioned other computing machine any.

Computer system 400 can also comprise the software element shown for being currently located in working storage 418, comprise operating system 420 and/or other code 422, such as application program (it can be client application, Web browser, middle-tier application, RDBMS etc.).In the exemplary embodiment, working storage 418 can comprise executable code for above-mentioned technology and the data structure (such as high-speed cache) that is associated.Will be appreciated that, the alternative of computer system 400 can have and is multiplely different from above-described change.Such as, the hardware of customization also can be used and/or specific element can with hardware, software (comprising portable software, such as applet (small routine)) or both realizations.In addition, the connection of other computing equipment of such as network input-output apparatus can be used.

Storage medium and computer-readable medium for comprising code or code section can comprise known in this area or by any suitable medium used, comprise storage medium and communication media, such as, but not limited to, for such as computer-readable instruction, data structure, the volatibility that the storage of the information of program module or other data and so on and/or any method of transmission or technology realize and non-volatile, removable and non-removable medium, comprise RAM, ROM, EEPROM, flash memories or other memory technology, CD-ROM, numeral variation dish (DVD) or other optical memory, tape cassete, tape, disk memory or other magnetic storage apparatus, data-signal, data are transmitted, or can be used to store or send expectation information and can by other medium any of computer access.

Fig. 7 is the simplified block diagram of electronic equipment 700 according to an embodiment of the invention.Electronic equipment 700 can comprise start unit 702, and it is configured to start first operation performing the first inquiry relative to the event data be stored in by the very first time in persistent data memory block; Performance element 704, it is configured to, when first operation performs, perform the second inquiry continuously relative to the event data received from data stream continuously; Merge cells 706, it is configured to, when first operation performs, the result that the second result of inquiring about and completed second batch operate be merged continuously; Output unit 708, it is configured to the result exporting merging continuously; And replacement unit 710, it is configured to completing in response to first operation, replaces the result of second batch operation by the result of first operation.

In the example shown, electronic equipment 700 also comprises storage unit, and it is configured to when first operation performs, continuously and in persistent data memory block, store the event data received from data stream continuously; Second start unit, it is configured to complete startup the 3rd batch operation in response to first operation, and the 3rd batch operation performs the first inquiry relative to the event data that has been stored in by the second time being later than the very first time in persistent data memory block; Second performance element, it is configured to, when the 3rd batch operation performs, perform the second inquiry continuously relative to the event data received from data stream continuously; Second merge cells, it is configured to when the 3rd batch operation performs, and the second result of inquiring about and the completed result that first operates is merged continuously; Second output unit, it is configured to export continuously the result that the second inquiry and the completed result that first operates merge; And second replacement unit, it is configured to completing in response to the 3rd batch operation, replaces the result of first operation by the result of the 3rd batch operation.

In the example shown, first operation maps reduction operation, and wherein the second query execution inquires about identical operation with first.

In the example shown, electronic equipment 700 also comprises receiving element, and it is configured to receive continuous-query language (CQL) inquiry as the second inquiry; And generation unit, it is configured to the mapping reduction program automatically generating the operation performing CQL inquiry based on CQL inquiry.

In the example shown, electronic equipment 700 also comprises: the second receiving element, and it is configured to receive continuous-query language (CQL) inquiry as the second inquiry; Resolution unit, it is configured to automatically resolve CQL inquiry; Separative element, it is configured to automatically CQL inquiry is separated into first group of operational symbol based on parsing; Second generation unit, it is configured to automatically generate based on the mapping of specifying between CQL operational symbol and Pig Latin operational symbol the second group of Pig Latin operational symbol performing the operational symbol performed by first group of CQL operational symbol; And the 3rd generation unit, it is configured to automatically generate based on Pig Latin operational symbol mapping reduction program performing first operation.

In the example shown, electronic equipment 700 also comprises: the 3rd receiving element, and it is configured to receive continuous-query language (CQL) inquiry as the second inquiry; Second resolution unit, it is configured to automatically resolve CQL inquiry; Second separative element, it is configured to automatically CQL inquiry is separated into first group of operational symbol based on parsing; 4th generation unit, it is configured to automatically generate based on the mapping of specifying between CQL operational symbol and Pig Latin operational symbol the second group of Pig Latin operational symbol performing the operational symbol performed by first group of CQL operational symbol; And the 5th generation unit, it is configured to the digraph automatically generating node, this digraph comprise from first group operational symbol, from the operational symbol of second group, from the node represented from the operational symbol of first group to representing from the limit of the node of the operational symbol of second group and from the node represented from the operational symbol of second group to the limit of the node represented from the operational symbol of first group.

Various unit disclosed herein can use hardware, software or its combination realize or perform.They can realize by general single-chip or multi-chip processor, digital signal processor (DSP), special IC (ASIC), field programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components or its combination in any being designed to perform function described herein.General processor can be microprocessor, or, the processor of any routine, controller, microcontroller or state machine.Processor can also be embodied as the combination of computing equipment, such as, and the combination of DSP and microprocessor, multi-microprocessor, the one or more microprocessor be combined with DSP kernel or other this configuration any.In some implementations, unit can be realized by the Circuits System specific to given function.

Although describe specific embodiments of the invention, various amendment, change, constructive alternative and equivalent are also contained within scope of the present invention.Amendment, change, constructive alternative and equivalent comprise the combination relevant arbitrarily of disclosed feature.Embodiments of the invention are not limited to the operation in some concrete data processing circumstance, but can freely operate in several data processing environment.In addition, although embodiments of the invention have utilized the particular sequence of affairs and step to be described, for a person skilled in the art should obviously, scope of the present invention is not limited to described affairs and the sequence of step.

In addition, although embodiments of the invention have utilized the particular combination of hardware and software to be described, will be appreciated that, other combination of hardware and software also within the scope of the invention.Embodiments of the invention can only with hardware or only with software or utilize its combination to realize.

Therefore, this instructions and accompanying drawing will be understood in illustrative instead of restrictive meaning.But, obviously, when do not deviate from as in claim set forth spirit and scope widely, can add it, reduce, delete and other amendment and change.

Claims (20)

1. a computer implemented method, comprising:

Start first operation, it performs the first inquiry relative to the event data be stored in by the very first time in persistent data memory block;

When first operation performs, perform the second inquiry continuously relative to the event data received from data stream continuously;

When first operation performs, continuously the result that the second result of inquiring about and completed second batch operate is merged;

Export the result of merging continuously; And

In response to completing of first operation, replace the result of second batch operation by the result of first operation.

2. computer implemented method as claimed in claim 1, also comprises:

When first operation performs, in persistent data memory block, store the event data received from data stream continuously continuously;

In response to completing of first operation, start the 3rd batch operation, it performs the first inquiry relative to the event data be stored in by the second time being later than the very first time in persistent data memory block;

When the 3rd batch operation performs, perform the second inquiry continuously relative to the event data received from data stream continuously;

When the 3rd batch operation performs, continuously the second result of inquiring about and the completed result that first operates are merged;

Export the result that the second inquiry merges with the completed result that first operates continuously; And

In response to completing of the 3rd batch operation, replace the result of first operation by the result of the 3rd batch operation.

3., as computer implemented method according to claim 1 or claim 2, wherein first operation maps reduction operation, and wherein the second query execution inquires about identical operation with first.

4., as the computer implemented method in claims 1 to 3 as described in any one, also comprise:

Receive continuous-query language (CQL) inquiry as the second inquiry; And

Based on CQL inquiry, automatically generate the mapping reduction program of the operation performing CQL inquiry.

5., as the computer implemented method in Claims 1-4 as described in any one, also comprise:

Receive continuous-query language (CQL) inquiry as the second inquiry;

Automatically resolve CQL inquiry;

Based on parsing, automatically CQL inquiry is separated into first group of operational symbol;

Based on the mapping of specifying between CQL operational symbol and Pig Latin operational symbol, automatically generate the second group of Pig Latin operational symbol performing the operational symbol performed by first group of CQL operational symbol; And

Based on Pig Latin operational symbol, automatically generate the mapping reduction program performing first operation.

6., as the computer implemented method in claim 1 to 5 as described in any one, also comprise:

Receive continuous-query language (CQL) inquiry as the second inquiry;

Automatically resolve CQL inquiry;

Based on parsing, automatically CQL inquiry is separated into first group of operational symbol;

Based on the mapping of specifying between CQL operational symbol and Pig Latin operational symbol, automatically generate the second group of Pig Latin operational symbol performing the operational symbol performed by first group of CQL operational symbol; And

Automatically generate the digraph of node, this digraph comprise from first group operational symbol, from the operational symbol of second group, from the node represented from the operational symbol of first group to representing from the limit of the node of the operational symbol of second group and from the node represented from the operational symbol of second group to the limit of the node represented from the operational symbol of first group.

7., as the computer implemented method in claim 1 to 6 as described in any one, also comprise:

Receive continuous-query language (CQL) inquiry as the second inquiry;

Automatically resolve CQL inquiry;

Based on parsing, automatically CQL inquiry is separated into first group of operational symbol;

Based on the mapping of specifying between CQL operational symbol and Pig Latin operational symbol, automatically generate the second group of Pig Latin operational symbol performing the operational symbol performed by first group of CQL operational symbol;

Automatically generate the digraph of node, this digraph comprises the operational symbol from first group and the operational symbol from second group; And

Based on the similarity between the operation performed by two or more nodes in digraph, automatically merge this two or more nodes.

8. a computer-readable memory, it stores the specific instruction making one or more processor executable operations, and described specific instruction comprises:

Make one or more processor start the instruction of first operation, wherein first operation performs the first inquiry relative to the event data be stored in by the very first time in persistent data memory block;

When first operation performs, make one or more processor relative to the event data received from data stream continuously and perform continuously the instruction of the second inquiry;

When first operation performs, the instruction of the result merging that the second result of inquiring about and completed second batch are operated by one or more processor continuously;

One or more processor is made to export the instruction of the result of merging continuously; And

What one or more processor was operated in response to first completes, and replaces the instruction of the result of second batch operation by the result of first operation.

9. computer-readable memory as claimed in claim 8, wherein specific instruction also comprises:

When first operation performs, make one or more processor in persistent data memory block, store the instruction of the event data received from data stream continuously continuously;

What one or more processor was operated in response to first completes, and starts the instruction of the 3rd batch operation, and wherein the 3rd batch operation performs the first inquiry relative to the event data be stored in by the second time being later than the very first time in persistent data memory block;

When the 3rd batch operation performs, make one or more processor relative to the event data received from data stream continuously and perform continuously the instruction of the second inquiry;

When the 3rd batch operation performs, make one or more processor continuously by the second result of inquiring about and the instruction that merges of the completed result that first operates;

One or more processor is made to export the instruction of the result that the second inquiry and the completed result that first operates merge continuously; And

Make one or more processor completing in response to the 3rd batch operation, replace the instruction of the result of first operation by the result of the 3rd batch operation.

10. computer-readable memory as claimed in claim 8 or claim 9, wherein first operation maps reduction operation, and wherein the second query execution inquires about identical operation with first.

11. as the computer-readable memory in claim 8 to 10 as described in any one, and wherein specific instruction also comprises:

One or more processor is made to receive the instruction of continuous-query language (CQL) inquiry as the second inquiry; And

One or more processor is inquired about based on this CQL, automatically generates the instruction of the mapping reduction program of the operation performing CQL inquiry.

12. as the computer-readable memory in claim 8 to 11 as described in any one, and wherein specific instruction also comprises:

One or more processor is made to receive the instruction of continuous-query language (CQL) inquiry as the second inquiry;

One or more processor is made automatically to resolve the instruction of CQL inquiry;

Make one or more processor based on parsing, automatically CQL inquiry is separated into the instruction of first group of operational symbol;

Make one or more processor based on the mapping of specifying between CQL operational symbol and Pig Latin operational symbol, automatically generate the instruction of the second group of Pig Latin operational symbol performing the operational symbol performed by first group of CQL operational symbol; And

Make one or more processor based on Pig Latin operational symbol, automatically generate the instruction of the mapping reduction program performing first operation.

13. as the computer-readable memory in claim 8 to 12 as described in any one, and wherein specific instruction also comprises:

One or more processor is made to receive the instruction of continuous-query language (CQL) inquiry as the second inquiry;

One or more processor is made automatically to resolve the instruction of CQL inquiry;

Make one or more processor based on parsing, automatically CQL inquiry is separated into the instruction of first group of operational symbol;

Make one or more processor based on the mapping of specifying between CQL operational symbol and Pig Latin operational symbol, automatically generate the instruction of the second group of Pig Latin operational symbol performing the operational symbol performed by first group of CQL operational symbol; And

Make one or more processor automatically generate the instruction of the digraph of node, wherein digraph comprise from first group operational symbol, from the operational symbol of second group, from the node represented from the operational symbol of first group to representing from the limit of the node of the operational symbol of second group and from the node represented from the operational symbol of second group to the limit of the node represented from the operational symbol of first group.

14. as the computer-readable memory in claim 8 to 13 as described in any one, and wherein specific instruction also comprises:

One or more processor is made to receive the instruction of continuous-query language (CQL) inquiry as the second inquiry;

One or more processor is made automatically to resolve the instruction of CQL inquiry;

Make one or more processor based on parsing, automatically CQL inquiry is separated into the instruction of first group of operational symbol;

Make one or more processor based on the mapping of specifying between CQL operational symbol and Pig Latin operational symbol, automatically generate the instruction of the second group of Pig Latin operational symbol performing the operational symbol performed by first group of CQL operational symbol;

Make one or more processor automatically generate the instruction of the digraph of node, wherein digraph comprises the operational symbol from first group and the operational symbol from second group; And

Make one or more processor based on the similarity between the operation performed by two or more nodes in digraph, automatically merge the instruction of these two or more nodes.

15. 1 kinds of systems, comprising:

One or more processor; And

Store the computer-readable memory of specific instruction, wherein specific instruction makes one or more processor executable operations, and specific instruction comprises:

Make one or more processor start the instruction of first operation, wherein first operation performs the first inquiry relative to the event data be stored in by the very first time in persistent data memory block;

When first operation performs, make one or more processor relative to the event data received from data stream continuously and perform continuously the instruction of the second inquiry;

When first operation performs, the instruction of the result merging that the second result of inquiring about and completed second batch are operated by one or more processor continuously;

One or more processor is made to export the instruction of the result of merging continuously; And

What one or more processor was operated in response to first completes, and replaces the instruction of the result of second batch operation by the result of first operation.

16. systems as claimed in claim 15, wherein specific instruction also comprises:

When first operation performs, make one or more processor in persistent data memory block, store the instruction of the event data received from data stream continuously continuously;

What one or more processor was operated in response to first completes, and starts the instruction of the 3rd batch operation, and wherein the 3rd batch operation performs the first inquiry relative to the event data be stored in by the second time being later than the very first time in persistent data memory block;

When the 3rd batch operation performs, make one or more processor relative to the event data received from data stream continuously and perform continuously the instruction of the second inquiry;

When the 3rd batch operation performs, make one or more processor continuously by the second result of inquiring about and the instruction that merges of the completed result that first operates;

One or more processor is made to export the instruction of the result that the second inquiry and the completed result that first operates merge continuously; And

Make one or more processor completing in response to the 3rd batch operation, replace the instruction of the result of first operation by the result of the 3rd batch operation.

17. as claim 15 or system according to claim 16, and wherein first operation maps reduction operation, and wherein the second query execution inquires about identical operation with first.

18. as the system in claim 15 to 17 as described in any one, and wherein specific instruction also comprises:

One or more processor is made to receive the instruction of continuous-query language (CQL) inquiry as the second inquiry; And

One or more processor is inquired about based on this CQL, automatically generates the instruction of the mapping reduction program of the operation performing CQL inquiry.

19. as the system in claim 15 to 18 as described in any one, and wherein specific instruction also comprises:

One or more processor is made to receive the instruction of continuous-query language (CQL) inquiry as the second inquiry;

One or more processor is made automatically to resolve the instruction of CQL inquiry;

Make one or more processor based on parsing, automatically CQL inquiry is separated into the instruction of first group of operational symbol;

Make one or more processor based on the mapping of specifying between CQL operational symbol and Pig Latin operational symbol, automatically generate the instruction of the second group of Pig Latin operational symbol performing the operational symbol performed by first group of CQL operational symbol; And

Make one or more processor based on Pig Latin operational symbol, automatically generate the instruction of the mapping reduction program performing first operation.

20. as the system in claim 15 to 19 as described in any one, and wherein specific instruction also comprises:

One or more processor is made to receive the instruction of continuous-query language (CQL) inquiry as the second inquiry;

One or more processor is made automatically to resolve the instruction of CQL inquiry;

Make one or more processor based on parsing, automatically CQL inquiry is separated into the instruction of first group of operational symbol;

Make one or more processor based on the mapping of specifying between CQL operational symbol and Pig Latin operational symbol, automatically generate the instruction of the second group of Pig Latin operational symbol performing the operational symbol performed by first group of CQL operational symbol; And

Make one or more processor automatically generate the instruction of the digraph of node, wherein digraph comprise from first group operational symbol, from the operational symbol of second group, from the node represented from the operational symbol of first group to representing from the limit of the node of the operational symbol of second group and from the node represented from the operational symbol of second group to the limit of the node represented from the operational symbol of first group.