CN105204822A - Multiple data stream processing method based on MIC co-processor - Google Patents
Multiple data stream processing method based on MIC co-processor Download PDFInfo
- Publication number
- CN105204822A CN105204822A CN201510707467.4A CN201510707467A CN105204822A CN 105204822 A CN105204822 A CN 105204822A CN 201510707467 A CN201510707467 A CN 201510707467A CN 105204822 A CN105204822 A CN 105204822A
- Authority
- CN
- China
- Prior art keywords
- data
- window
- data stream
- mic
- stream
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a multiple data stream processing method based on an MIC co-processor. The method comprises the steps that a central processing unit (CPU) acquires a data source and transmits multiple data streams which arrive in parallel to the MIC co-processor through a PCIE bus; the MIC co-processor performs parallel calculation on the data streams by adopting a four-layer sliding window model and carries out a multiple data stream query algorithm to obtain a data stream query result; the MIC co-processor returns the data stream query result to the CPU; the CPU obtains an IO output task of a user according to the data stream query result and carries out the IO output task. According to the multiple data stream processing method based on the MIC co-processor, the parallel performance and real-time performance of multiple data stream processing are improved.
Description
Technical field
The present invention relates to the technical field of large data high-performance calculation, particularly relate to a kind of multi-data-stream processing method based on MIC coprocessor.
Background technology
In actual life, express network fault diagnosis, the Trade Data Stream in retail trade, online auction, transaction log, Web follow the tracks of and define a kind of data shape different from static data in traditional database with the image data stream etc. that Monitoring Data and the satellite of the telephony recording data stream in personalization, medical monitoring, the communications field, the data packet stream in network monitor, environment temperature are passed back.Data in data stream arrive be fast, time become, unpredictable and unlimited data-stream form, raw data can not be stored completely.And the data volume that these data stream produce increases fast in multiple application, and the application producing data stream requires online process in real time usually.
For traditional techniques for Multiple Data-Streams Processing basic model, traditional techniques for Multiple Data-Streams Processing technology by all deposit data in database or data warehouse; The DML statement that system responses user submits to, search data storage medium, returns Query Result.When data scale is very large, data often with disk or tape for medium, thus performing query manipulation needs a large amount of I/O to exchange, and inefficiency, can not adapt to the demand of real-time system.Constantly change due to multiple data stream itself and be difficult to the feature of prediction, and the generation of multiple data stream burst is had higher requirement to multiple data stream load capacity, techniques for Multiple Data-Streams Processing is difficult to because time overhead is excessive meet real-time demand, so the concurrency of techniques for Multiple Data-Streams Processing and real-time are very low simultaneously.
Summary of the invention
The object of this invention is to provide a kind of multi-data-stream processing method based on MIC coprocessor, to realize the concurrency and the real-time that improve techniques for Multiple Data-Streams Processing.
For solving the problems of the technologies described above, the invention provides the multi-data-stream processing method based on MIC coprocessor, the method comprises:
Central processor CPU obtains data source, many data of parallel arrival is flowed through PCIE bus transfer in MIC coprocessor;
Described MIC coprocessor adopts four layers of sliding window model to carry out parallel computation to described many data streams, and performs multiple data stream search algorithm, obtains Data stream query result;
Described Data stream query result is back to described CPU by described MIC coprocessor;
The IO that described CPU obtains user according to described Data stream query result exports task, performs described IO and exports task.
Preferably, described central processor CPU also comprises after obtaining data source:
Many data stream of parallel arrival are pooled to front buffer zone by described central processor CPU, and buffer contents before and after exchanging, by the exchanges data of buffer window in rear buffer area in MIC coprocessor.
Preferably, described four layers of sliding window model comprise: time series data layer, buffer window layer, moving window layer and summary data matrix.
Preferably, the data of described time series data layer are tlv triple: <SID, Timestamp, Value>, SID are stream identification, and Timestamp is tuple time of arrival, Value is data value, described time series data layer is used for data IO process and data buffering process, and to the data calculating with identical SID arrived in arbitrary data sampling time unit and value, what do not have data to arrive processes according to 0 interpolation or linear interpolation.
Preferably, the data of described buffer window layer are four-tuple: <SID, Timestamp, Data, Synopsis>, SID is data flow identifiers, Timestamp is the timestamp of data sharing in all basic windows, Data is the set of the data in window, the statistical information of this buffer window of Synopsis or summary info, described buffer window layer be used for by PCIE bus by the exchanges data of buffer window corresponding for every bar data stream in the internal memory of MIC coprocessor, MIC coprocessor generates the summary info of basic window.
Preferably, the data of described moving window layer are tlv triple: <SID, Data, Synopsis>, SID represents the data flow identifiers of this moving window, Data is sliding window data, sliding window data is the set of the data being a continuous print w/b basic window in physical store, Synopsis is the summary data of whole moving window, described moving window layer is used for when new buffer window is switched to the internal memory of MIC coprocessor, the overall summary info of incremental maintenance every bar data stream.
Preferably, described summary data matrix is: M=[s
0, s
1... s
n-1]
t, M is described summary matrix, s
ithe row vector of the data of the data stream of index SID=i or summary data structure composition, described summary data matrix be used for by by multiple moving window dense arrangement in summary data matrix, be kept in continuous print memory headroom.
Preferably, described four layers of sliding window model are four layers of sliding window model across PCIE bus.
Preferably, described MIC coprocessor performs multiple data stream search algorithm by minimum exchange principle, only returns last Query Result.
Preferably, described central processor CPU obtains data source, after many data of parallel arrival being flowed through in PCIE bus transfer to MIC coprocessor, also comprises:
Many data stream are stored in continuous print memory headroom by described MIC coprocessor, utilize the read-write operation that index identifier and side-play amount walk abreast to described many data stream.
A kind of multi-data-stream processing method based on MIC coprocessor provided by the present invention, central processor CPU obtains data source, many data of parallel arrival is flowed through PCIE bus transfer in MIC coprocessor; Described MIC coprocessor adopts four layers of sliding window model to carry out parallel computation to described many data streams, and performs multiple data stream search algorithm, obtains Data stream query result; Described Data stream query result is back to described CPU by described MIC coprocessor; The IO that described CPU obtains user according to described Data stream query result exports task, performs described IO and exports task.Visible, data are directly carried out read-write process by the method, do not need in write into Databasce, do not need with disk to be that medium is inquired about yet, save the whole parallel processing time, and MIC coprocessor executed in parallel is transferred in the parallel computation process of multiple data stream and search algorithm, utilize central processing unit process complicated process flow process and powerful data buffering ability, focus on the task of Row control and the thousands of data stream of data buffering, realize the concurrency and the real-time that improve techniques for Multiple Data-Streams Processing like this.
Accompanying drawing explanation
In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, be briefly described to the accompanying drawing used required in embodiment or description of the prior art below, apparently, accompanying drawing in the following describes is only embodiments of the invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to the accompanying drawing provided.
Fig. 1 is the process flow diagram of a kind of multi-data-stream processing method based on MIC coprocessor provided by the invention;
Fig. 2 is four layers of sliding window model schematic diagram;
Fig. 3 is the techniques for Multiple Data-Streams Processing schematic flow sheet of MIC coprocessor.
Embodiment
Core of the present invention is to provide a kind of multi-data-stream processing method based on MIC coprocessor, to realize the concurrency and the real-time that improve techniques for Multiple Data-Streams Processing.
The present invention program is understood better in order to make those skilled in the art person, below in conjunction with the accompanying drawing in the embodiment of the present invention, technical scheme in the embodiment of the present invention is clearly and completely described, obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.
Please refer to Fig. 1, Fig. 1 is the process flow diagram of a kind of multi-data-stream processing method based on MIC coprocessor provided by the invention, and the method comprises:
S11: central processor CPU obtains data source, many data of parallel arrival is flowed through PCIE bus transfer in MIC coprocessor;
Wherein, in fact data stream is exactly the element troop of continuous moving, and element is wherein made up of the set of related data.Make t represent arbitrary timestamp, at represents the data arrived at this timestamp, flow data can be expressed as ..., at-1, at, at+1 ....Be different from traditional application model, stream data model has following general character: data arrive in real time; It is independent that data arrive order, do not controlled by application system; Data scale is grand and can not predict its maximal value; Data one are treated, unless specially preserved, otherwise again can not be taken out process, or extraction data cost dearly again.
After central processor CPU obtains data source, many data stream of parallel arrival are pooled to front buffer zone by central processor CPU, and buffer contents before and after exchanging, by the exchanges data of buffer window in rear buffer area in MIC coprocessor.
After many data of parallel arrival flow through in PCIE bus transfer to MIC coprocessor by central processor CPU, many data stream are stored in continuous print memory headroom by MIC coprocessor, utilize the read-write operation that index identifier and side-play amount walk abreast to described many data stream.
S12:MIC coprocessor adopts four layers of sliding window model to carry out parallel computation to many data streams, and performs multiple data stream search algorithm, obtains Data stream query result;
Wherein, MIC (ManyIntegratedCores, integrated many-core processor) be coprocessor based on X86-based, there is operating system and quite high memory bandwidth, MIC comprises 61 and calculates core, eachly endorse support 4 hardware threads, CPU can be helped to carry out the evaluation work of some complexity, MIC can be regarded as an independently small server.In programming novel aspects, MIC adopts compiling to instruct the mode of statement to carry out, and difficulty is little, and the construction cycle is short, is easy to safeguard.Perform multiple data stream search algorithm, inquire about many data streams, what obtain is Query Result, i.e. Data stream query result.
The integrated many nuclear technology of MIC are using CPU as main frame, and MIC is as the collaborative work of both coprocessors.CPU is responsible for carrying out the strong transaction of logicality and serial computing, and MIC is then absorbed in the highly threading parallel processing task of execution.CPU, MIC coprocessor has separate memory address space separately: host side internal memory and MIC coprocessor internal memory.Once determine the parallel computation function in program, just consider this part calculating to give MIC.
Described four layers of sliding window model comprise: time series data layer, buffer window layer, moving window layer and summary data matrix.MIC coprocessor performs most according to multiple data stream search algorithm by minimum exchange principle, only return last Query Result.Described four layers of sliding window model are four layers of sliding window model across PCIE bus.
Data stream query result is back to CPU by S13:MIC coprocessor;
The IO that S14:CPU obtains user according to Data stream query result exports task, performs IO and exports task.
For multiple data stream, four layers of sliding window model provided by the invention are followed minimum data and are exchanged principle, across PCIE bus.MIC coprocessor has significant advantage than pure CPU process in the real-time of multiple data stream related coefficient accurate Calculation.The framework of these four layers of sliding window models can be generalized to multiple data stream and excavates in other research fields, as multiple data stream principal component calculates and multiple data stream cluster analysis, has certain versatility.
For four layers of sliding window model, concrete, the data of time series data layer are tlv triple: <SID, Timestamp, Value>, SID is stream identification, Timestamp is tuple time of arrival, and Value is data value, and time series data layer is used for data IO process and data buffering process, calculate and value the data with identical SID arrived in any data sampling time unit, what do not have data to arrive processes according to 0 interpolation or linear interpolation.
The data of buffer window layer are four-tuple: <SID, Timestamp, Data, Synopsis>, SID is data flow identifiers, Timestamp is the timestamp of data sharing in all basic windows, Data is the set of the data in window, the statistical information of this buffer window of Synopsis or summary info, buffer window layer be used for by PCIE bus by the exchanges data of buffer window corresponding for every bar data stream in the internal memory of MIC coprocessor, MIC coprocessor generates the summary info of basic window.
The data of moving window layer are tlv triple: <SID, Data, Synopsis>, SID represents the data flow identifiers of this moving window, Data is sliding window data, sliding window data is the set of the data being a continuous print w/b basic window in physical store, Synopsis is the summary data of whole moving window, moving window layer is used for when new buffer window is switched to the internal memory of MIC coprocessor, the overall summary info of incremental maintenance every bar data stream.
Summary data matrix is: M=[s
0, s
1... s
n-1]
t, M is summary matrix, s
ithe row vector of the data of the data stream of index SID=i or summary data structure composition, summary data matrix be used for by by multiple moving window dense arrangement in summary data matrix, be kept in continuous print memory headroom.
In the concrete use of the method, need the Framework system of hardware system and four layers of sliding window model, wherein, hardware system comprises: a MIC high-performance server system, this system node adopts CPU+MIC isomery framework, in node except cpu chip, also containing at least one MIC coprocessor.
The transmission of data between CPU main frame and MIC coprocessor device needs through PCIE bus, although PCIE2.O has the theoretical bandwidth of 3.2G/s, transmitting data between still needs extra clock expense.In the application of MIC coprocessor data streams in parallel process, exchanges data is inevitable frequently, and this is also the unfavorable factor that MIC coprocessor data streams in parallel calculates.And exchanges data frequency therebetween and exchanges data amount can be reduced across four layers of sliding window model of PCIE bus.
Mic card can be regarded as a coprocessor of CPU, and it plays optimized algorithm working time thus improves Data Stream Processing real-time, and the dual role of sharing data stream Processing tasks.In four layers of sliding window model, MIC coprocessor utilizes mining algorithm parallel-parallel data streams, central processing unit has process complicated process flow process and the ability of powerful data buffering, focuses on the task of Row control and the thousands of data stream of data buffering.
Concrete, in the Framework system of four layers of sliding window model, based on four layers of sliding window model of basic window method, its most outstanding feature has three kinds of varigrained chronomeres: data sampling time unit, basic window chronomere and moving window chronomere.And.The important meaning of of basic window is reduction of the Time & Space Complexity safeguarded multiple moving window.At MIC coprocessor in the process of parallel data stream, basic window improves Data Stream Processing parallelization granularity.
Data sampling time unit is the minimum time unit of image data, and the data stream tuple arrived in this time period has identical timestamp.The tentation data sampling time, unit was Δ t, and current time stamp is t, then the timestamp of the arbitrary data arrived after t can be expressed as t+i × Δ t, wherein i ∈ I.
Basic window chronomere is the base unit safeguarding moving window, any renewal moving window, Delete Expired basic window, and the maintenance of statistics, summary info is all carried out with this chronomere.Basic window chronomere is less, and the real-time of Data Stream Processing is higher, and the computing cost safeguarded data stream is larger.If each basic window contains b element, then the time span of each basic window is Δ T=b × Δ t.The size of basic window chronomere needs to determine according to the actual processing capability in real time of practical application needs and data stream.Although less basic window chronomere can provide better Data Stream Processing real-time.
For the basic window method that four layers of gliding model adopt, basic window method is a kind of when computing power deficiency, expands the technology of Data Stream Processing and mining algorithm time interval granularity.At MIC coprocessor in the process of parallel data stream, what sometimes occur calculates because data stream piecemeal is not enough to meet MIC coprocessor total power, cause the situation of performance loss, and in this case, basic window method enables MIC coprocessor carry out parallel computation with higher efficiency to data stream.
Moving window chronomere is the data stream time span that practical application needs to safeguard.If moving window length is w, number of windows is n, and basic window size is b, then moving window length is w=n × b.
Be propose on basis based on the sliding window model of basic window technology across four layers of sliding window model of PCIE bus, as shown in Figure 2, Fig. 2 is four layers of sliding window model schematic diagram.The data of time series data layer are tlv triple: <SID, Timestamp, Value>, SID are stream identification, and Timestamp is tuple time of arrival, and Value is data value.Task mainly data IO and the data buffering of this layer.In this one deck, the data with identical SID arrived in any data sampling time unit get itself and, there is no data arrival table then by 0 interpolation or linear interpolation processing.Such as: the data stream for stock exchange hand number should adopt 0 interpolation, and the data stream of value of stocks is then more suitable for linear interpolation.The calculation task mainly data buffering of time series data layer, namely data are obtained from data source, and SID and Timestamp is considered as index, only its value is write be added to corresponding SID and Timestamp in multiple data stream data buffer position on, until write the data of a full basic window.Because CPU has better surge capability than MIC coprocessor, therefore the calculation task of this layer transfers to CPU to complete completely.
In MIC coprocessor techniques for Multiple Data-Streams Processing model, for cushioning the data of a basic window amount of capacity, exchanging a basic window of data to MIC coprocessor device, being called buffer window.The essence of buffer window is a basic window, and it is basic window specific appellation in a particular state, and buffer window, for reducing the exchanges data frequency between main frame and coprocessor, improves the dense degree of multiple data stream parallel computation.Data buffering layer is the least unit of MIC coprocessor to multiple data stream parallel processing, is updated in moving window as common subwindow through plucking buffer window to be processed.Data buffering layer adopts random memory access mechanism, meets MIC parallel computation to the specification requirement of data structure and the requirement of data parallel access efficiency.
The data of buffer window layer are four-tuple a: <SID, Timestamp, Data, Synopsis>, wherein, SID is data flow identifiers, and Timestamp is the timestamp of data sharing in all basic windows, Data is the set of the data in window, the statistical information of this buffer window of Synopsis or summary info.Difference according to application can omitted data Data, and only retains its summary info Synopsis.Calculation task from buffer window is transferred on MIC coprocessor, MIC coprocessor uses MIC accounting method executed in parallel calculation task, the calculation task of moving window layer has two: by PCIE bus by the exchanges data of n buffer window of n bar data stream to MIC internal memory, and on MIC coprocessor, perform basic window summary info safeguard kernel, generate the summary info of basic window.Through MIC coprocessor kernel processes and the buffer window be updated on moving window, a common basic window can be considered as, the maintenance of the accrual accounting information of buffer window on basic window.
The data of moving window layer are tlv triple: <SID, Data, Synopsis>, wherein, SID represents the data flow identifiers of this moving window, Data is sliding window data, and it is the set of the data of a continuous print w/b basic window in physical store, and Synopsis is the summary data of whole moving window.Moving window is identical with basic window, can select whether retain window data as required.
Moving window layer is the set of all not out of date basic windows in moving window, if the length of moving window is w, then for { t-w+1, data in the t} time, during renewal due to moving window in units of basic window, if the length of each basic window is b, then for any data stream, have b data expired, b new data is updated at every turn.The task of moving window layer is when new buffer window is switched to MIC internal memory, the overall summary info of incremental maintenance every bar data stream.
Summary data matrix is that the moving window of w and summary info thereof are organized in continuous print memory headroom respectively by n bar length, and make its data-intensive arrangement, formation can unify index, the matrix of concurrent access.This matrix can be expressed as M=[s
0, s
1... s
n-1]
t.Wherein s
ithe data of the data stream of SID=i or the row vector of summary data structure composition.By by n bar moving window dense arrangement in summary data matrix, and be kept in continuous print memory headroom, make this data structure support random memory access mechanism, and meet MIC parallel computation to the specification requirement of data structure and the requirement of data parallel access efficiency.
In parallel computation, by by data abstraction in MIC thread, the calculating that the Multi-core of MIC coprocessor walks abreast to data can be used.Present invention achieves a kind of multi-data-stream processing method based on MIC coprocessor, the method utilizes MIC coprocessor to carry out parallel computation to multiple data stream in a more efficient manner, thus improves the processing capability in real time of data stream.Data streams in parallel processing procedure herein realizes on MIC coprocessor.
Please refer to Fig. 3, Fig. 3 is the techniques for Multiple Data-Streams Processing schematic flow sheet of MIC coprocessor.The main task division of labor is according to being the real-time needs that minimum data between CPU main frame and MIC coprocessor device exchanges principle and Data Stream Processing.This schematic diagram characterizes three layers of concept: techniques for Multiple Data-Streams Processing task is divided the work, data flow and main functional modules.The requirement of real-time Data Stream Processing of data stream should complete within the time short as far as possible, and use MIC coprocessor parallel data processing stream is the real-time in order to improve Data Stream Processing, and by PCIE bus switch between main frame and equipment.Data there will be again extra IO expense, although to reach 3.2G in theory per second for PCIE bus bandwidth height, for the high request of real-time, this IO expense still should reduce as much as possible, to optimize the processing procedure that multiple data stream excavates.A kind of multi-data-stream processing method based on MIC coprocessor that the present invention proposes, datastream source will be obtained, data bufferings etc. need the operation of complicated Row control and data buffering to transfer to central processing unit to complete, and the Data Stream Processing algorithm of intensive parallel computation can transfer to the parallel computation of MIC coprocessor, the IO finally Data stream query result being returned to user exports task and also transfers to CPU main frame to complete, and this just as shown in Figure 3.
Concrete, be first that input and buffering: CPU are responsible for obtaining data source, and many data stream of parallel arrival are pooled to front window buffer zone; Buffer contents before and after exchanging; Products for further in the exchanges data of a rear buffer zone n buffer window to MIC coprocessor device internal memory is calculated.
Next is kernel and the summary data structure division of parallel computation: in MIC parallel programming model, what utilize the C language of expansion to write is called by main frame, MIC coprocessor performs, and what use #pragmaoffloadtarget (mic) compiling to instruct statement to state is called kernel program.
Summary class kernel is responsible for generating and incremental maintenance buffer window and moving window summary info; Inquiry class kernel is responsible for performing query task; Excavate kernel with summary data matrix for input performs multiple data stream mining algorithm.Summary data structure is kept on MIC coprocessor internal memory with the form of summary data matrix.
Because the method uses summary data matrix to carry out dense tissue and unified index to many data streams, in the window matrix making the method can be generalized to any use multi-slide-windows mouth composition or the multiple data stream mining algorithm of increment window matrix as input.Only just need can complete corresponding data Mining stream task by writing different multiple data stream excavations and inquiring about MIC coprocessor kernel program, there is stronger versatility.
Finally output: follow the minimum data exchange principle that MIC coprocessor universal parallel calculates, the multiple data stream search algorithm that MIC coprocessor performs, should only return last Query Result, exchange on CPU main frame, and via CPU, Query Result is returned to client.
To sum up, a kind of multi-data-stream processing method based on MIC coprocessor provided by the present invention, central processor CPU obtains data source, many data of parallel arrival is flowed through PCIE bus transfer in MIC coprocessor; MIC coprocessor adopts four layers of sliding window model to carry out parallel computation to many data streams, and performs multiple data stream search algorithm, obtains Data stream query result; Data stream query result is back to CPU by MIC coprocessor; The IO that CPU obtains user according to Data stream query result exports task, performs IO and exports task.Visible, data are directly carried out read-write process by the method, do not need in write into Databasce, do not need with disk to be that medium is inquired about yet, save the whole parallel processing time, and MIC coprocessor executed in parallel is transferred in the parallel computation process of multiple data stream and search algorithm, utilize central processing unit process complicated process flow process and powerful data buffering ability, focus on the task of Row control and the thousands of data stream of data buffering, realize the concurrency and the real-time that improve techniques for Multiple Data-Streams Processing like this.
Above a kind of multi-data-stream processing method based on MIC coprocessor provided by the present invention is described in detail.Apply specific case herein to set forth principle of the present invention and embodiment, the explanation of above embodiment just understands method of the present invention and core concept thereof for helping.It should be pointed out that for those skilled in the art, under the premise without departing from the principles of the invention, can also carry out some improvement and modification to the present invention, these improve and modify and also fall in the protection domain of the claims in the present invention.
Claims (10)
1. based on a multi-data-stream processing method for MIC coprocessor, it is characterized in that, comprising:
Central processor CPU obtains data source, many data of parallel arrival is flowed through PCIE bus transfer in MIC coprocessor;
Described MIC coprocessor adopts four layers of sliding window model to carry out parallel computation to described many data streams, and performs multiple data stream search algorithm, obtains Data stream query result;
Described Data stream query result is back to described CPU by described MIC coprocessor;
The IO that described CPU obtains user according to described Data stream query result exports task, performs described IO and exports task.
2. the method for claim 1, is characterized in that, described central processor CPU also comprises after obtaining data source:
Many data stream of parallel arrival are pooled to front buffer zone by described central processor CPU, and buffer contents before and after exchanging, by the exchanges data of buffer window in rear buffer area in MIC coprocessor.
3. the method for claim 1, is characterized in that, described four layers of sliding window model comprise: time series data layer, buffer window layer, moving window layer and summary data matrix.
4. method as claimed in claim 3, it is characterized in that, the data of described time series data layer are tlv triple: <SID, Timestamp, Value>, SID is stream identification, Timestamp is tuple time of arrival, Value is data value, described time series data layer is used for data IO process and data buffering process, calculate and value the data with identical SID arrived in any data sampling time unit, what do not have data to arrive processes according to 0 interpolation or linear interpolation.
5. method as claimed in claim 3, it is characterized in that, the data of described buffer window layer are four-tuple: <SID, Timestamp, Data, Synopsis>, SID is data flow identifiers, Timestamp is the timestamp of data sharing in all basic windows, Data is the set of the data in window, the statistical information of this buffer window of Synopsis or summary info, described buffer window layer be used for by PCIE bus by the exchanges data of buffer window corresponding for every bar data stream in the internal memory of MIC coprocessor, MIC coprocessor generates the summary info of basic window.
6. method as claimed in claim 3, it is characterized in that, the data of described moving window layer are tlv triple: <SID, Data, Synopsis>, SID represents the data flow identifiers of this moving window, Data is sliding window data, sliding window data is the set of the data being a continuous print w/b basic window in physical store, Synopsis is the summary data of whole moving window, described moving window layer is used for when new buffer window is switched to the internal memory of MIC coprocessor, the overall summary info of incremental maintenance every bar data stream.
7. method as claimed in claim 3, it is characterized in that, described summary data matrix is: M=[s
0, s
1... s
n-1]
t, M is described summary matrix, s
ithe row vector of the data of the data stream of index SID=i or summary data structure composition, described summary data matrix be used for by by multiple moving window dense arrangement in summary data matrix, be kept in continuous print memory headroom.
8. the method for claim 1, is characterized in that, described four layers of sliding window model are four layers of sliding window model across PCIE bus.
9. the method for claim 1, is characterized in that, described MIC coprocessor performs multiple data stream search algorithm by minimum exchange principle, only returns last Query Result.
10. method as in one of claimed in any of claims 1 to 9, is characterized in that, described central processor CPU obtains data source, after many data of parallel arrival being flowed through in PCIE bus transfer to MIC coprocessor, also comprises:
Many data stream are stored in continuous print memory headroom by described MIC coprocessor, utilize the read-write operation that index identifier and side-play amount walk abreast to described many data stream.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510707467.4A CN105204822A (en) | 2015-10-27 | 2015-10-27 | Multiple data stream processing method based on MIC co-processor |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510707467.4A CN105204822A (en) | 2015-10-27 | 2015-10-27 | Multiple data stream processing method based on MIC co-processor |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105204822A true CN105204822A (en) | 2015-12-30 |
Family
ID=54952534
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510707467.4A Pending CN105204822A (en) | 2015-10-27 | 2015-10-27 | Multiple data stream processing method based on MIC co-processor |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105204822A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109388609A (en) * | 2018-09-30 | 2019-02-26 | 中科驭数（北京）科技有限公司 | Based on the data processing method and device for accelerating core |
| CN109542662A (en) * | 2018-11-23 | 2019-03-29 | 北京锐安科技有限公司 | A kind of EMS memory management process, device, server and storage medium |
| CN112799603A (en) * | 2021-03-02 | 2021-05-14 | 王希敏 | Task behavior model for multiple data stream driven signal processing system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103440229A (en) * | 2013-08-12 | 2013-12-11 | 浪潮电子信息产业股份有限公司 | Vectorizing optimization method based on MIC (Many Integrated Core) architecture processor |
| US8893103B2 (en) * | 2012-08-16 | 2014-11-18 | Nec Laboratories America, Inc. | Automatic asynchronous offload to many-core coprocessors |
| US20150277877A1 (en) * | 2014-03-25 | 2015-10-01 | Nec Laboratories America, Inc. | Compiler optimization for many integrated core processors |
-
2015
- 2015-10-27 CN CN201510707467.4A patent/CN105204822A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8893103B2 (en) * | 2012-08-16 | 2014-11-18 | Nec Laboratories America, Inc. | Automatic asynchronous offload to many-core coprocessors |
| CN103440229A (en) * | 2013-08-12 | 2013-12-11 | 浪潮电子信息产业股份有限公司 | Vectorizing optimization method based on MIC (Many Integrated Core) architecture processor |
| US20150277877A1 (en) * | 2014-03-25 | 2015-10-01 | Nec Laboratories America, Inc. | Compiler optimization for many integrated core processors |
Non-Patent Citations (4)
| Title |
|---|
| 周勇: "基于GPU 的多数据流相关系数并行计算方法研究", 《计算机应用研究》 * |
| 周勇: "基于并行计算的数据流处理方法研究", 《中国优秀博士学位论文全文数据库信息科技辑》 * |
| 沈铂: "基于ＭＩＣ平台的ｏｆｆｌｏａｄ并行方法研究", 《计算机科学》 * |
| 王寅峰: "MIC商用并行编程性能优化分析", 《深圳信息职业技术学院学报》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109388609A (en) * | 2018-09-30 | 2019-02-26 | 中科驭数（北京）科技有限公司 | Based on the data processing method and device for accelerating core |
| CN109542662A (en) * | 2018-11-23 | 2019-03-29 | 北京锐安科技有限公司 | A kind of EMS memory management process, device, server and storage medium |
| CN109542662B (en) * | 2018-11-23 | 2022-04-05 | 北京锐安科技有限公司 | Memory management method, device, server and storage medium |
| CN112799603A (en) * | 2021-03-02 | 2021-05-14 | 王希敏 | Task behavior model for multiple data stream driven signal processing system |
| CN112799603B (en) * | 2021-03-02 | 2024-05-14 | 王希敏 | Task behavior model for multi-data stream driven signal processing system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110741356B (en) | Relay coherent memory management in multiprocessor systems | |
| CN104391679A (en) | GPU (graphics processing unit) processing method for high-dimensional data stream in irregular stream | |
| CN108280522A (en) | A kind of plug-in type distributed machines study Computational frame and its data processing method | |
| CN102227121B (en) | Distributed buffer memory strategy adaptive switching method based on machine learning and system thereof | |
| CN101650698B (en) | Method for realizing direct memory access | |
| CN105590321B (en) | A kind of block-based subgraph building and distributed figure processing method | |
| Xu et al. | Temperature aware workload managementin geo-distributed data centers | |
| Oliveira et al. | Trends in computation, communication and storage and the consequences for data-intensive science | |
| CN102307206B (en) | Caching system and caching method for rapidly accessing virtual machine images based on cloud storage | |
| Kleiminger et al. | Balancing load in stream processing with the cloud | |
| CN110245178A (en) | Marketing automation management platform system and its management method | |
| CN103401777A (en) | Parallel search method and system of Openflow | |
| CN106503276A (en) | A kind of method and apparatus of the time series databases for real-time monitoring system | |
| Mazzucco et al. | Balancing electricity bill and performance in server farms with setup costs | |
| CN105204822A (en) | Multiple data stream processing method based on MIC co-processor | |
| CN111639054B (en) | Data coupling method, system and medium for ocean mode and data assimilation | |
| CN102567079B (en) | Parallel program energy consumption simulation estimating method based on progressive trace update | |
| Frey et al. | Spinning relations: high-speed networks for distributed join processing | |
| CN105989065A (en) | Flash shot data processing method and system | |
| CN102497419A (en) | ESB (Enterprise Service Bus) task scheduling method orienting to large-scale requests and scheduling system | |
| CN110059024A (en) | A kind of memory headroom data cache method and device | |
| Astsatryan et al. | Performance-efficient recommendation and prediction service for Big Data frameworks focusing on data compression and in-memory data storage indicators | |
| CN115713395A (en) | Flink-based user wind control management method, device and equipment | |
| CN104899007A (en) | System and method for improving processing performances of Bloom filter through utilizing Xeon Phi coprocessor | |
| CN103377085A (en) | Method, device and system for instruction management and operation core |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20151230 |
|
| RJ01 | Rejection of invention patent application after publication |