CN108985709A - Workflow management method towards more satellite data centers collaboration Remote Sensing Products production - Google Patents
Workflow management method towards more satellite data centers collaboration Remote Sensing Products production Download PDFInfo
- Publication number
- CN108985709A CN108985709A CN201810671454.XA CN201810671454A CN108985709A CN 108985709 A CN108985709 A CN 108985709A CN 201810671454 A CN201810671454 A CN 201810671454A CN 108985709 A CN108985709 A CN 108985709A
- Authority
- CN
- China
- Prior art keywords
- remote sensing
- data
- task
- workflow
- sensing products
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
- G06Q10/103—Workflow collaboration or project management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06393—Score-carding, benchmarking or key performance indicator [KPI] analysis
Landscapes
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Engineering & Computer Science (AREA)
- Strategic Management (AREA)
- Entrepreneurship & Innovation (AREA)
- Economics (AREA)
- General Business, Economics & Management (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Development Economics (AREA)
- Educational Administration (AREA)
- Data Mining & Analysis (AREA)
- Game Theory and Decision Science (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a kind of Workflow management methods towards more satellite data centers collaboration Remote Sensing Products production, comprising the following steps: building Remote Sensing Products produce process flow library, and the process knowledge base of building multi- source Remote Sensing Data data processing constructs multistage task order store；Construct the varigrained specific tasks workflow of Remote Sensing Products production automatically according to Kepler Dynamic Workflow System and PBS job scheduling system；Formulate heuristic workflow Dynamic Scheduling Strategy method；The strategy that relevant multistage task status monitoring is produced to Remote Sensing Products is formulated, and based on the fault-tolerant strategy retried, checkpoint is restored, time-out exits；Construct can dynamic process library, algorithms library, realize the dynamic scalable management of more satellite data center multi-source remote sensing workflows.
Description
Technical field
The present invention relates to satellite data Center Technology fields, it particularly relates to which a kind of assist towards more satellite data centers
With the Workflow management method of Remote Sensing Products production.
Background technique
Currently, many countries and regions have been set up different remote sensing satellite platform and satellite data center, have
Multispectral, multi-angle, multidate, the spatial observation of more spatial resolutions and data-handling capacity.These different types of satellites
Platform has generated and by the lasting remotely-sensed data for generating magnanimity, to meet the information extraction and application demand of specialization.Due to
Single satellite data source is difficult to meet the integrated application demand of remote sensing fields large size application, by current multiple satellite data centers
Mechanism joins together, and provides the satellite remote sensing date support of multi-source, magnanimity, carries out large-scale data processing and inversion, meet
User's comprehensive remote sensing information service demand has become one of the important trend of current remote sensing application.
However, combining the distributed remotely-sensed data coprocessing system of multiple satellite data center constructions, there are technologies to ask
Topic.Firstly, multi- source Remote Sensing Data data is distributed in different satellite data centers, the large-scale Data Migration of production process influences most
Processing whole efficiency is cooperateed with according to center；Secondly, the complicated stream such as Remote Sensing Data Processing itself is pre-processed, post-processing
Journey, and there are greatest differences between the process flow of multi- source Remote Sensing Data data, to increase the complexity of production work flow management
Property；In addition, comprehensive remote sensing applies the data processing task for needing processing system that can be automatically performed batch toward contact, in distribution
Realize that Complicated Flow automatic management is also to influence one of the critical issue of multiple data centers coprocessing system building under scene.
For magnanimity Remote Sensing Products production requirement and the major issue of multiple data centers collaboration complex process process automation management, currently
Solution is difficult to the complication system framework according to more satellite data centers, systematically makes management, the work of suitable process
Make task automation tissue, scheduling, expansion management strategy.
Summary of the invention
For above-mentioned technical problem in the related technology, the present invention proposes a kind of towards the collaboration remote sensing of more satellite data centers
The Workflow management method of production can overcome the above-mentioned deficiency of the prior art.
To realize the above-mentioned technical purpose, the technical scheme of the present invention is realized as follows:
A kind of Workflow management method towards more satellite data centers collaboration Remote Sensing Products production, comprising the following steps:
S1: model conclusion is carried out to the usual process flow of Remote Sensing Products production, building Remote Sensing Products produce process flow
Library, for the single process in process library, difference and Remote Sensing Products for multi-source remote sensing product rely on the difference of data source, building
The process knowledge base of multi- source Remote Sensing Data data processing, according to the multistep treatment process in process library, dynamic construction is more in process of production
Grade task order store；
S2: multi-layer, distributed system features for more satellite data centring systems, according to Kepler dynamic duty
Streaming system and PBS job scheduling system construct the varigrained specific tasks workflow of Remote Sensing Products production automatically；
S3: for more satellite data centers collaboration the data-intensive, computation-intensive of Remote Sensing Data Processing, distributed computing,
The level feature of the characteristics of multiple-objection optimization and workflow, formulate comprising data transmission, task queue and calculated performance based on
The heuristic workflow Dynamic Scheduling Strategy method of more satellite data center collaboration Remote Sensing Products production scenes；
S4: according to specific automation, the Remote Sensing Products production environment of batch, multistage relevant to Remote Sensing Products production is formulated
The strategy of task status monitoring, and based on the fault-tolerant strategy retried, checkpoint is restored, time-out exits；
S5: updating rapid feature for multi-source remote sensing product manufacturing process, algorithm, construct can dynamic process library,
Algorithms library realizes the dynamic scalable management of more satellite data center multi-source remote sensing workflows.
Further, in step S1, described Remote Sensing Products production process flow library include Remote Sensing Products production classification and
Its corresponding sub-process.
Preferably, Remote Sensing Products production classification includes remote sensing general character product, RS fusion product and remote sensing assimilation
Product；The sub-process includes pretreatment process and post-processing process.
Further, in step S1, the process knowledge base of the multi- source Remote Sensing Data data processing refers to multi- source Remote Sensing Data data
Produce the different step that same Remote Sensing Products are included, specifically include the data corrected need not move through geometry essence correction,
Some data need not move through radiation correction and Standard division range product needs not move through Standard division range.
Preferably, the process knowledge base of multi- source Remote Sensing Data data processing further includes that different Remote Sensing Products are relied on not
Same data source.
Further, in step S1, the multistage task order store refers to raw with Remote Sensing Products in the actual production process
Produce process library and multi- source Remote Sensing Data data processing process knowledge base corresponding to specific production procedure, according to different process steps,
The location of task rank and data source, the multistage production task order that dynamic generates, the multistage production task order
Automation building, monitoring and scheduling for production procedure.
Further, step S2 is specifically included:
S2.1: constructing according to the automation that the workflow template of Kepler dynamic duty system carries out production procedure, should be certainly
Dynamicization building process relies on Remote Sensing Products production procedure library and carries out the process knowledge base of multi- source Remote Sensing Data data processing, matches process
It is automated；
S2.2: being based on PBS job scheduling system method for auto constructing, produces task feature in batches according to practical, formulates system
One PBS operating system template, realizes the unified management of batch tasks.
Further, in step S3, the inspiration based on more satellite data centers collaboration Remote Sensing Products production scene
Formula workflow Dynamic Scheduling Strategy method specifically:
Estimated based on mass data transfers time Estimate, each satellite data central task queue situation in process of producing product
The task processing capacity at meter and each satellite data center calculates three aspect factor, in conjunction with system monitoring and task order status structure
Prediction model is built, resource matched and task schedule is carried out.
Preferably, resource matched and task schedule to comprise the concrete steps that:
S3.1: it is dispatched and is established to execute the objective function of time-constrain based on Best-effort, workflow schedule target is
The execution time of one three-level task order is most short, regulation goal function are as follows:
ECT (t, r)=max { EAT (t, r), FAT (t, r) }+EET (t, r)
Wherein, wherein t is a three-level order taking responsibility, and r is resource collection required for three-level order, and ECT (t, r) is three
The Estimated Time Of Completion of grade order taking responsibility, EET (t, r) be after task schedule in current data supercentral task execution estimation
Between, EAT (t, r) is the estimation time that current data center can execute task t, and FAT (t, r) is resource r all in current number
According to the estimation time ready on center；
S3.2: data resource scheduling: buffered in preferential distribution remotely-sensed data caching library by data management system
Data resource needs to download according to data type to corresponding data center requests for uncached data resource, according to asking
Data volume is asked to estimate the FAT (t, r) of each data center；
S3.3: computing resource scheduling: monitoring the performance information of each data center by deployment resource monitoring dynamic,
By CPU, network, load information real-time update into computing resource library, in scheduling phase, believed according to the monitoring of each data center
The calculated performance at prediction data center is ceased, and carries out EET (t, r) estimation；
S3.4: task queue calculates: by each data center PBS task queue of multistage task order library inquiry and appointing
Task run status predication can run the time of current task in business order store, and carry out EAT (t, r) estimation；
S3.5: it is based on the principle construction of " nearly data calculating " look_ahead schedule, is contained heuristic for dispatching
Rule.
Preferably, in step S3.5, for dispatching the empirical parameter that didactic rule includes estimation FAT, when EET estimates
The weight parameter of performance indexes, Performance Counter Threshold when scheduling of resource.
Beneficial effects of the present invention: the present invention is devised according to the Remote Sensing Products production environment that more satellite data centers cooperate with
It is suitble to the Workflow management method of master-slave mode distributed structure/architecture, by constructing Remote Sensing Products process flow library, multi- source Remote Sensing Data data
The process knowledge base of processing realizes that the Complicated Flow management of multi- source Remote Sensing Data data product and complex data rely on management, is corresponding to it
Multistage task order store can fast implement the rapid build of specific production workflow, can so significantly simplify Remote Sensing Products
The model of production and realization, the scheduling rule library based on multiple-objection optimization realize the dynamic dispatching pipe of production workflow
Reason, farthest matches suitable data resource and computing resource, to greatly improve more satellite data centers cooperating manufacture
Performance, process flow library, algorithms library and workflow template library substantially increase the scalability of production work flow management.
Detailed description of the invention
It in order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, below will be to institute in embodiment
Attached drawing to be used is needed to be briefly described, it should be apparent that, the accompanying drawings in the following description is only some implementations of the invention
Example, for those of ordinary skill in the art, without creative efforts, can also obtain according to these attached drawings
Obtain other attached drawings.
Fig. 1 is the extensive Remote Sensing Products production work of the more satellite data centers collaboration described according to embodiments of the present invention
Workflow Management System integrated stand composition；
Fig. 2 is that Remote Sensing Products production workflow management operating timing is illustrated on primary data center described in the embodiment of the present invention
Figure；
Fig. 3 is that task execution timing is illustrated in distributed satellites data center production system described in the embodiment of the present invention
Figure；
Fig. 4 a and Fig. 4 b be production process flow tree in Remote Sensing Products process flow library described in the embodiment of the present invention and
Multistep treatment task order store schematic diagram；
Fig. 5 is the Remote Sensing Products dynamic dispatching illustraton of model described in the embodiment of the present invention based on heuristic rule.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, those of ordinary skill in the art's every other embodiment obtained belong to what the present invention protected
Range.
As shown in Figure 1, described one kind is raw towards more satellite data centers collaboration Remote Sensing Products according to embodiments of the present invention
The Workflow management method of production, comprising the following steps:
S1: model conclusion is carried out to the usual process flow of Remote Sensing Products production, building Remote Sensing Products produce process flow
Library, for the single process in process library, difference and Remote Sensing Products for multi-source remote sensing product rely on the difference of data source, building
The process knowledge base of multi- source Remote Sensing Data data processing, according to the multistep treatment process in process library, dynamic construction is more in process of production
Grade task order store；
S2: multi-layer, distributed system features for more satellite data centring systems, according to Kepler dynamic duty
Streaming system and PBS job scheduling system construct the varigrained specific tasks workflow of Remote Sensing Products production automatically；
S3: for more satellite data centers collaboration the data-intensive, computation-intensive of Remote Sensing Data Processing, distributed computing,
The level feature of the characteristics of multiple-objection optimization and workflow, formulate comprising data transmission, task queue and calculated performance based on
The heuristic workflow Dynamic Scheduling Strategy method of more satellite data center collaboration Remote Sensing Products production scenes；
S4: according to specific automation, the Remote Sensing Products production environment of batch, multistage relevant to Remote Sensing Products production is formulated
The strategy of task status monitoring, and based on the fault-tolerant strategy retried, checkpoint is restored, time-out exits；
S5: updating rapid feature for multi-source remote sensing product manufacturing process, algorithm, construct can dynamic process library,
Algorithms library realizes the dynamic scalable management of more satellite data center multi-source remote sensing workflows.
In one embodiment, in step S1, the Remote Sensing Products production process flow library includes that Remote Sensing Products are raw
Produce classification and its corresponding sub-process.
In one embodiment, Remote Sensing Products production classification includes remote sensing general character product, RS fusion product
Assimilate product with remote sensing；The sub-process includes pretreatment process and post-processing process.
In one embodiment, in step S1, the process knowledge base of the multi- source Remote Sensing Data data processing refers to multi-source
Remotely-sensed data produces the different step that same Remote Sensing Products are included, and specifically includes the data corrected and needs not move through geometry
Essence correction, some data need not move through radiation correction and Standard division range product needs not move through Standard division range.
In one embodiment, the process knowledge base of multi- source Remote Sensing Data data processing further includes different Remote Sensing Products
The different data sources relied on.
In one embodiment, in step S1, the multistage task order store refer in the actual production process with it is distant
Specific production procedure corresponding to the process knowledge base of product manufacturing process library and multi- source Remote Sensing Data data processing is felt, according to not cocurrent flow
The location of journey step, task rank and data source, the multistage production task order that dynamic generates, the multistage production are appointed
Automation building, monitoring and scheduling of the business order for production procedure.
In one embodiment, step S2 is specifically included:
S2.1: constructing according to the automation that the workflow template of Kepler dynamic duty system carries out production procedure, should be certainly
Dynamicization building process relies on Remote Sensing Products production procedure library and carries out the process knowledge base of multi- source Remote Sensing Data data processing, matches process
It is automated；
S2.2: being based on PBS job scheduling system method for auto constructing, produces task feature in batches according to practical, formulates system
One PBS operating system template, realizes the unified management of batch tasks.
In one embodiment, described to produce field based on more satellite data centers collaboration Remote Sensing Products in step S3
The heuristic workflow Dynamic Scheduling Strategy method of scape specifically:
Estimated based on mass data transfers time Estimate, each satellite data central task queue situation in process of producing product
The task processing capacity at meter and each satellite data center calculates three aspect factor, in conjunction with system monitoring and task order status structure
Prediction model is built, resource matched and task schedule is carried out.
In one embodiment, resource matched and task schedule to comprise the concrete steps that:
S3.1: it is dispatched and is established to execute the objective function of time-constrain based on Best-effort, workflow schedule target is
The execution time of one three-level task order is most short, regulation goal function are as follows:
ECT (t, r)=max { EAT (t, r), FAT (t, r) }+EET (t, r)
Wherein, wherein t is a three-level order taking responsibility, and r is resource collection required for three-level order, and ECT (t, r) is three
The Estimated Time Of Completion of grade order taking responsibility, EET (t, r) be after task schedule in current data supercentral task execution estimation
Between, EAT (t, r) is the estimation time that current data center can execute task t, and FAT (t, r) is resource r all in current number
According to the estimation time ready on center；
S3.2: data resource scheduling: buffered in preferential distribution remotely-sensed data caching library by data management system
Data resource needs to download according to data type to corresponding data center requests for uncached data resource, according to asking
Data volume is asked to estimate the FAT (t, r) of each data center；
S3.3: computing resource scheduling: monitoring the performance information of each data center by deployment resource monitoring dynamic,
By CPU, network, load information real-time update into computing resource library, in scheduling phase, believed according to the monitoring of each data center
The calculated performance at prediction data center is ceased, and carries out EET (t, r) estimation；
S3.4: task queue calculates: by each data center PBS task queue of multistage task order library inquiry and appointing
Task run status predication can run the time of current task in business order store, and carry out EAT (t, r) estimation；
S3.5: it is based on the principle construction of " nearly data calculating " look_ahead schedule, is contained heuristic for dispatching
Rule.
Wherein, in step S3.5, for dispatching the empirical parameter that didactic rule includes estimation FAT, EET is each when estimating
The weight parameter of performance indicator, Performance Counter Threshold when scheduling of resource.
In order to facilitate understanding above-mentioned technical proposal of the invention, below by way of in specifically used mode to of the invention above-mentioned
Technical solution is described in detail.
A kind of Workflow Management side towards more satellite data centers collaboration Remote Sensing Products production according to the present invention
Method, including following basic module and module:
1) multiple data centers collaboration Remote Sensing Products production framework includes a primary data center and multiple distributed satellites
Data center, and entire Workflow system is constructed based on this.The process flow library of production is constructed on primary data center,
The process knowledge base of multi- source Remote Sensing Data data processing, and multistage task is established in primary data center and distributed satellites data center
Order store be used for Remote Sensing Products production procedure unified management, and realize global calculation task order split, condition monitoring with
Feedback；
2) the process flow library in workflow automatic constructing module, primary data center, workflow template library, task order
Library etc. can be realized the tissue of the Auto-matching of complex process process, Task-decomposing and Abstract workflow, based on Kepler science
Workflow system, to realize the automatic building of Abstract workflow to Concrete workflow；
3) in workflow dynamic dispatching module, by what is unified on primary data center to algorithm resource and computing resource
Management, provides information service for workflow schedule.System integrally uses two-level scheduler strategy on Scheduling Framework, in master data
The heart utilizes the performance monitoring information and task load information of each data center, and calculating task is dispatched to each data center,
Processing Mission Operations management and scheduling are realized based on Torque PBS scheduling system inside data center；
4) in workflow monitoring and fault-tolerant module, each distributed satellites data center is actively anti-to the product of primary data center
Task status is presented, the execution state of meeting query task during the Kepler workflow execution of primary data center, when midway is malfunctioned,
The strategies such as system can be retried according to scheduled fault-tolerant strategy, checkpoint is restored, time-out exits, carry out production workflow
It is fault-tolerant, until work flow operation terminates.
When specific more satellite data center collaboration Remote Sensing Products productions, Remote Sensing Products production work flow tube on primary data center
Reason operation timing as shown in Fig. 2, carry out automatic building, the work of Abstract workflow building, task schedule, Concrete workflow respectively
Stream verification with execute, workflow status monitor with it is fault-tolerant.Task execution timing is such as in distributed satellites data center production system
Shown in Fig. 3, task receives, the automatically generating of PBS operation order, the batch of PBS algorithm script are submitted, the monitoring of PBS order and holds
Fault reason, the feedback of task order status.
When specifically used, 1), production processing Task Tree and multistep treatment task order store
Generation method such as Fig. 4 institute of production processing Task Tree and multistep treatment task order store that the present invention uses
Show, the original data type Auto-matching multi- source Remote Sensing Data data pretreatment process library according to needed for production and different remote sensing numbers
According to process flow knowledge base, after the completion of matching, the Remote Sensing Products processing stream of a task tree is established in flow of task library
Journey is as shown in fig. 4 a.Need to carry out the preliminary treatment of task order, order data parsing, order production after Task Tree construction complete really
Recognize, the matching of production procedure, the processing task such as the fractionation of multistage order Auto-matching decompose after, in primary data center multistage
Multistage task order corresponding with process flow library is formd in order store, as shown in Figure 4 b, multistage task order precedence relationship,
Belonging relation is just defined, and the abstract workflow of production process flow is formd.
2), Remote Sensing Products production Concrete workflow constructs automatically
In the present invention, the automatic building of the Concrete workflow of primary data center is needed by the multistage order generated
Library and Concrete workflow based on Kepler workflow construct template automatically, in the template, corresponding multistage task order store, and example
Such as the work stream file that second level order is unit, this document includes multiple process flows, and each process flow is mainly one three
Grade order, each three-level order mainly include the logics such as task is submitted, executes status inquiry, task status is fed back step by step, work
Stream can automatically update error condition by branching logic, and then stop working stream, avoid error-logic.Secondly, will own
Actors is connected, and is grouped according to functional module, and the workflow template of entire three-level order is formd.Finally, can be by
According to the process of Abstract workflow, the Kepler workflow template of arbitrary three-level order is combined, workflow template is formd.
3), two-stage Remote Sensing Products production task dynamic dispatching frame
Based on heuristic rule be that workflow task selects suitable data center resource on primary data center, mainly with
Lower step is completed: 1, the foundation of objective function: it is dispatched and is established to execute the objective function of time-constrain based on Best-effort,
Workflow schedule target is that the execution time of a three-level task order is most short, and regulation goal function is
ECT (t, r)=max { EAT (t, r), FAT (t, r) }+EET (t, r)
Wherein t is a three-level order taking responsibility, and r is resource collection required for three-level order, predominantly distant in this system
Feeling data, ECT (t, r) is the Estimated Time Of Completion (Estimated Completion Time, ECT) of three-level order taking responsibility,
EET (t, r) is to estimate time (Estimated Execution in the supercentral task execution of current data after task schedule
Time, EET), major influence factors are data center's calculated performance, and EAT (t, r) is that current data center can execute task t
The estimation time (Estimated Availability Time, EAT), as long as image factors be current data center task
Queue, FAT (t, r) are resource r all ready estimation time (File Available on current data center
Time, FAT), major influence factors are transmitting data resources.Objective function considers data when each computing resource executes task
Time FAT, the task queue deadline EAT before dispatching, current order task execution time EET；2, data resource
Scheduling: it is preferential to distribute data resource buffered in remotely-sensed data caching library by data management system, for uncached
Data resource needs to download according to data type to corresponding data center requests, estimates each data according to request data quantity
The FAT (t, r) at center；3, computing resource is dispatched: the performance of each data center is monitored by deployment resource monitoring dynamic
Information, by the information such as CPU, IO, Mem, network, load real-time update into computing resource library, in scheduling phase, according to every number
According to the calculated performance at the monitoring information prediction data center at center, and carry out EET (t, r) estimation；4, task queue calculates: passing through
Task run status predication can be transported in each data center PBS task queue of multistage task order library inquiry and task order store
The time of row current task, and carry out EAT (t, r) estimation；5 in addition, the principle construction based on " nearly data calculating " is heuristic
Scheduling rule contains some for dispatching didactic rule, the empirical parameter including estimating FAT, items when EET estimates
Can index weight parameter, Performance Counter Threshold when scheduling of resource is based on scheduling rule library, can complete batch remotely-sensed data point
Cloth handles the automatic dispatching of task, and dynamic dispatching process is illustrated in fig. 5 shown below.
The scheduling strategy of the PBS task at each satellite data center is using prerequisite variable FIFO (First In First
Out then) and priority-based integrated dispatch strategy carries out at corresponding PBS algorithm according to the priority of job queue
Manage the execution of order.
According to described above, invention completes a set of extensive, distributed multi-source remote sensing production Workflow Management side
Method is primarily adapted for use in more satellite data centers collaboration processing environment of client/server.This method using unified workflow management, from
The workflow building of dynamicization, the reliable control of heuristic Multiobjective Optimal Operation, process and monitoring etc. are tactful, can greatly promote
Multiple data centers cooperate with multi-source remote sensing product systems service feature and production efficiency.
In conclusion the present invention devises suitable principal and subordinate according to the Remote Sensing Products production environment that more satellite data centers cooperate with
The Workflow management method of formula distributed structure/architecture passes through building Remote Sensing Products process flow library, the stream of multi- source Remote Sensing Data data processing
Journey knowledge base realizes the Complicated Flow management of multi- source Remote Sensing Data data product and complex data relies on management, and corresponding multistage is appointed
Business order store can fast implement the rapid build of specific production workflow, can so significantly simplify the mould of the production of Remote Sensing Products
Type and realization, the scheduling rule library based on multiple-objection optimization realize the dynamic dispatching management of production workflow, maximum journey
Degree ground matches suitable data resource and computing resource, to greatly improve the performance of more satellite data centers cooperating manufacture, locates
Reason process library, algorithms library and workflow template library substantially increase the scalability of production work flow management.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all in essence of the invention
Within mind and principle, any modification, equivalent replacement, improvement and so on be should all be included in the protection scope of the present invention.
Claims (10)
1. a kind of Workflow management method towards more satellite data centers collaboration Remote Sensing Products production, which is characterized in that including
Following steps:
S1: model conclusion is carried out to the usual process flow of Remote Sensing Products production, building Remote Sensing Products produce process flow library, right
Single process in process library, difference and Remote Sensing Products for multi-source remote sensing product rely on the difference of data source, construct multi-source
The process knowledge base of Remote Sensing Data Processing, according to the multistep treatment process in process library, dynamic construction multistage is appointed in process of production
Business order store；
S2: multi-layer, distributed system features for more satellite data centring systems, according to Kepler dynamic workflow system
System and PBS job scheduling system construct the varigrained specific tasks workflow of Remote Sensing Products production automatically；
S3: the data-intensive, computation-intensive of Remote Sensing Data Processing, distributed computing, more mesh are cooperateed with for more satellite data centers
The level feature of the characteristics of mark optimization and workflow, formulate comprising data transmission, task queue and calculated performance based on defending more
Sing data center cooperates with the heuristic workflow Dynamic Scheduling Strategy method of Remote Sensing Products production scene；
S4: according to specific automation, the Remote Sensing Products production environment of batch, multistage task relevant to Remote Sensing Products production is formulated
The strategy of condition monitoring, and based on the fault-tolerant strategy retried, checkpoint is restored, time-out exits；
S5: updating rapid feature for multi-source remote sensing product manufacturing process, algorithm, and constructing can dynamic process library, algorithm
Library realizes the dynamic scalable management of more satellite data center multi-source remote sensing workflows.
2. a kind of Workflow Management side towards more satellite data centers collaboration Remote Sensing Products production according to claim 1
Method, which is characterized in that in step S1, the Remote Sensing Products production process flow library includes Remote Sensing Products production classification and its phase
The sub-process answered.
3. a kind of Workflow Management side towards more satellite data centers collaboration Remote Sensing Products production according to claim 2
Method, which is characterized in that the Remote Sensing Products production classification includes that remote sensing general character product, RS fusion product and remote sensing assimilation produce
Product；The sub-process includes pretreatment process and post-processing process.
4. a kind of Workflow Management side towards more satellite data centers collaboration Remote Sensing Products production according to claim 1
Method, which is characterized in that in step S1, the process knowledge base of the multi- source Remote Sensing Data data processing refers to that multi- source Remote Sensing Data data produces
The different step that same Remote Sensing Products are included, specifically include the data corrected need not move through geometry essence correction, some
Data need not move through radiation correction and Standard division range product needs not move through Standard division range.
5. a kind of Workflow Management side towards more satellite data centers collaboration Remote Sensing Products production according to claim 4
Method, which is characterized in that the process knowledge base of the multi- source Remote Sensing Data data processing further includes that different Remote Sensing Products are relied on not
Same data source.
6. a kind of Workflow Management side towards more satellite data centers collaboration Remote Sensing Products production according to claim 1
Method, which is characterized in that in step S1, the multistage task order store, which refers to produce with Remote Sensing Products in the actual production process, to flow
Specific production procedure corresponding to the process knowledge base that Cheng Ku and multi- source Remote Sensing Data data are handled, according to different process steps, task
The location of rank and data source, the multistage production task order that dynamic generates, the multistage production task order are used for
Automation building, monitoring and the scheduling of production procedure.
7. a kind of Workflow Management side towards more satellite data centers collaboration Remote Sensing Products production according to claim 1
Method, which is characterized in that step S2 is specifically included:
S2.1: it is constructed according to the automation that the workflow template of Kepler dynamic duty system carries out production procedure, the automation
Building process relies on Remote Sensing Products production procedure library and carries out the process knowledge base of multi- source Remote Sensing Data data processing, and matching process carries out
Automation；
S2.2: being based on PBS job scheduling system method for auto constructing, produces task feature in batches according to practical, formulates unification
PBS operating system template, realizes the unified management of batch tasks.
8. a kind of Workflow Management side towards more satellite data centers collaboration Remote Sensing Products production according to claim 1
Method, which is characterized in that in step S3, the heuristic work based on more satellite data centers collaboration Remote Sensing Products production scene
Make flowable state scheduling strategy method specifically:
Based on mass data transfers time Estimate in process of producing product, the estimation of each satellite data central task queue situation and
The task processing capacity at each satellite data center calculates three aspect factor, pre- in conjunction with system monitoring and the building of task order status
Model is surveyed, resource matched and task schedule is carried out.
9. a kind of Workflow Management side towards more satellite data centers collaboration Remote Sensing Products production according to claim 8
Method, which is characterized in that resource matched and task schedule to comprise the concrete steps that:
S3.1: it is dispatched and is established to execute the objective function of time-constrain based on Best-effort, workflow schedule target is one
The execution time of three-level task order is most short, regulation goal function are as follows:
Wherein, wherein t is a three-level order taking responsibility, and r is resource collection required for three-level order, and ECT (t, r) orders for three-level
The Estimated Time Of Completion of single task, EET (t, r) are to estimate the time in the supercentral task execution of current data after task schedule,
EAT (t, r) is that current data center can execute estimation time of task t, and FAT (t, r) is resource r all in current data
The ready estimation time in the heart；
S3.2: data resource scheduling: preferential to distribute data buffered in remotely-sensed data caching library by data management system
Resource needs to download according to data type to corresponding data center requests, according to number of request for uncached data resource
The FAT (t, r) of each data center is estimated according to amount；
S3.3: computing resource scheduling: monitoring the performance information of each data center by deployment resource monitoring dynamic, will
CPU, network, load information real-time update are into computing resource library, in scheduling phase, according to the monitoring information of each data center
The calculated performance at prediction data center, and carry out EET (t, r) estimation；
S3.4: task queue calculates: being ordered by each data center PBS task queue of multistage task order library inquiry and task
Task run status predication can run the time of current task in single library, and carry out EAT (t, r) estimation；
S3.5: it is based on the principle construction of " nearly data calculating " look_ahead schedule, is contained for dispatching didactic rule
Then.
10. a kind of Workflow Management towards more satellite data centers collaboration Remote Sensing Products production according to claim 9
Method, which is characterized in that in step S3.5, for dispatching the empirical parameter that didactic rule includes estimation FAT, EET estimation
When performance indexes weight parameter, Performance Counter Threshold when scheduling of resource.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810671454.XA CN108985709A (en) | 2018-06-26 | 2018-06-26 | Workflow management method towards more satellite data centers collaboration Remote Sensing Products production |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810671454.XA CN108985709A (en) | 2018-06-26 | 2018-06-26 | Workflow management method towards more satellite data centers collaboration Remote Sensing Products production |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108985709A true CN108985709A (en) | 2018-12-11 |
Family
ID=64538833
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810671454.XA Pending CN108985709A (en) | 2018-06-26 | 2018-06-26 | Workflow management method towards more satellite data centers collaboration Remote Sensing Products production |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108985709A (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109657862A (en) * | 2018-12-20 | 2019-04-19 | 中国地质大学（武汉） | A kind of multi- source Remote Sensing Data data production workflow self-organizing method |
| CN109977036A (en) * | 2019-02-19 | 2019-07-05 | 东软集团股份有限公司 | Cache method, apparatus, storage medium and the electronic equipment of flow template |
| CN110941463A (en) * | 2019-11-13 | 2020-03-31 | 中国科学院遥感与数字地球研究所 | Remote sensing satellite data preprocessing multistage product self-driven system |
| CN111404593A (en) * | 2020-03-13 | 2020-07-10 | 北京华云星地通科技有限公司 | Processing method of satellite remote sensing data |
| CN111669213A (en) * | 2020-05-22 | 2020-09-15 | 军事科学院***工程研究院网络信息研究所 | Dynamic management and control system architecture and management and control method for satellite communication resources |
| CN111754073A (en) * | 2020-05-19 | 2020-10-09 | 北京吉威空间信息股份有限公司 | Centralized processing and distributed operation framework construction method for spatial data service |
| CN112231086A (en) * | 2020-10-22 | 2021-01-15 | 中国科学院空天信息创新研究院 | Production workflow description and scheduling method and device for remote sensing information product |
| CN112308443A (en) * | 2020-11-09 | 2021-02-02 | 中国科学院空天信息创新研究院 | Batch scheduling method and device for remote sensing information product generation workflow |
| CN112488492A (en) * | 2020-11-26 | 2021-03-12 | 中科星通(廊坊)信息技术有限公司 | Remote sensing product production scheduling method based on priority |
| CN112698859A (en) * | 2020-12-31 | 2021-04-23 | 中科星通(廊坊)信息技术有限公司 | Online product customization system and method based on remote sensing data |
| CN113010598A (en) * | 2021-04-28 | 2021-06-22 | 河南大学 | Dynamic self-adaptive distributed cooperative workflow system for remote sensing big data processing |
| CN114461357A (en) * | 2021-12-22 | 2022-05-10 | 中国科学院空天信息创新研究院 | Remote sensing satellite raw data real-time processing flow scheduling engine |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104484230A (en) * | 2014-12-31 | 2015-04-01 | 中国科学院遥感与数字地球研究所 | Multiple satellite data centre workflow scheduling algorithm on basis of near data calculation principle |
| CN105094982A (en) * | 2014-09-23 | 2015-11-25 | 航天恒星科技有限公司 | Multi-satellite remote sensing data processing system |
| CN106022245A (en) * | 2016-05-16 | 2016-10-12 | 中国资源卫星应用中心 | Multi-source remote sensing satellite data parallel processing system and method based on algorithm classification |
-
2018
- 2018-06-26 CN CN201810671454.XA patent/CN108985709A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105094982A (en) * | 2014-09-23 | 2015-11-25 | 航天恒星科技有限公司 | Multi-satellite remote sensing data processing system |
| CN104484230A (en) * | 2014-12-31 | 2015-04-01 | 中国科学院遥感与数字地球研究所 | Multiple satellite data centre workflow scheduling algorithm on basis of near data calculation principle |
| CN106022245A (en) * | 2016-05-16 | 2016-10-12 | 中国资源卫星应用中心 | Multi-source remote sensing satellite data parallel processing system and method based on algorithm classification |
Non-Patent Citations (2)
| Title |
|---|
| ZHANG, J., YAN, J., MA, Y. ET AL.: "《Infrastructures and services for remote sensing data production management across multiple satellite data centers》", 《CLUSTER COMPUT（2016）》 * |
| ZHANG,W.,WANG,L.,LIU,D.,SONG,W.,MA,Y.,LIU,P.,CHEN,D.: "《Towards building a multi-datacenter infrastructure for massive remote sensing image processing》", 《CONCURRENCY&COMPTUTATION PRACTICE&EXPERINECE》 * |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109657862A (en) * | 2018-12-20 | 2019-04-19 | 中国地质大学（武汉） | A kind of multi- source Remote Sensing Data data production workflow self-organizing method |
| CN109977036A (en) * | 2019-02-19 | 2019-07-05 | 东软集团股份有限公司 | Cache method, apparatus, storage medium and the electronic equipment of flow template |
| CN110941463A (en) * | 2019-11-13 | 2020-03-31 | 中国科学院遥感与数字地球研究所 | Remote sensing satellite data preprocessing multistage product self-driven system |
| CN111404593B (en) * | 2020-03-13 | 2022-02-15 | 北京华云星地通科技有限公司 | Method and device for processing satellite remote sensing data |
| CN111404593A (en) * | 2020-03-13 | 2020-07-10 | 北京华云星地通科技有限公司 | Processing method of satellite remote sensing data |
| CN111754073A (en) * | 2020-05-19 | 2020-10-09 | 北京吉威空间信息股份有限公司 | Centralized processing and distributed operation framework construction method for spatial data service |
| CN111754073B (en) * | 2020-05-19 | 2023-08-18 | 北京吉威空间信息股份有限公司 | Centralized processing and distributed operation framework construction method for space data service |
| CN111669213A (en) * | 2020-05-22 | 2020-09-15 | 军事科学院***工程研究院网络信息研究所 | Dynamic management and control system architecture and management and control method for satellite communication resources |
| CN112231086A (en) * | 2020-10-22 | 2021-01-15 | 中国科学院空天信息创新研究院 | Production workflow description and scheduling method and device for remote sensing information product |
| CN112231086B (en) * | 2020-10-22 | 2024-04-26 | 中国科学院空天信息创新研究院 | Method and device for describing and scheduling production workflow of remote sensing information product |
| CN112308443A (en) * | 2020-11-09 | 2021-02-02 | 中国科学院空天信息创新研究院 | Batch scheduling method and device for remote sensing information product generation workflow |
| CN112488492A (en) * | 2020-11-26 | 2021-03-12 | 中科星通(廊坊)信息技术有限公司 | Remote sensing product production scheduling method based on priority |
| CN112698859A (en) * | 2020-12-31 | 2021-04-23 | 中科星通(廊坊)信息技术有限公司 | Online product customization system and method based on remote sensing data |
| CN113010598B (en) * | 2021-04-28 | 2022-11-01 | 河南大学 | Dynamic self-adaptive distributed cooperative workflow system for remote sensing big data processing |
| CN113010598A (en) * | 2021-04-28 | 2021-06-22 | 河南大学 | Dynamic self-adaptive distributed cooperative workflow system for remote sensing big data processing |
| CN114461357A (en) * | 2021-12-22 | 2022-05-10 | 中国科学院空天信息创新研究院 | Remote sensing satellite raw data real-time processing flow scheduling engine |
| CN114461357B (en) * | 2021-12-22 | 2022-11-11 | 中国科学院空天信息创新研究院 | Remote sensing satellite original data real-time processing flow scheduling system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108985709A (en) | Workflow management method towards more satellite data centers collaboration Remote Sensing Products production | |
| Joo et al. | Multi-layered optimization of demand resources using lagrange dual decomposition | |
| CN103092683B (en) | For data analysis based on didactic scheduling | |
| US10474504B2 (en) | Distributed node intra-group task scheduling method and system | |
| CN101957780B (en) | Resource state information-based grid task scheduling processor and grid task scheduling processing method | |
| US20220027817A1 (en) | Deep reinforcement learning for production scheduling | |
| CN101237469B (en) | Method for optimizing multi-QoS grid workflow based on ant group algorithm | |
| CN105260818B (en) | Mix the on-line optimizing scheduling method with deadline constraint workflow group under cloud environment | |
| Liu et al. | Resource preprocessing and optimal task scheduling in cloud computing environments | |
| CN104731657B (en) | A kind of resource regulating method and system | |
| CN107015856A (en) | Task scheduling approach generation method and device under cloud environment in scientific workflow | |
| CN102752395B (en) | A kind of on-line scheduling method of distributing for distributive data center real time business | |
| CN106101196B (en) | A kind of cloud rendering platform task scheduling system based on probabilistic model | |
| CN109491761A (en) | Cloud computing multiple target method for scheduling task based on EDA-GA hybrid algorithm | |
| Durgadevi et al. | Resource allocation in cloud computing using SFLA and cuckoo search hybridization | |
| CN110739696A (en) | Integrated scheduling method for demand side resources and renewable energy in intelligent distribution network environment | |
| CN110928651B (en) | Service workflow fault-tolerant scheduling method under mobile edge environment | |
| Cao et al. | A parallel computing framework for large-scale air traffic flow optimization | |
| CN109559033B (en) | Socialized team member optimization method oriented to cloud design and manufacturing mode | |
| CN109615143A (en) | Wide-area Measurement Information management system task schedule ant colony optimization algorithm based on multi-QoS constraint | |
| Li et al. | An agent-based approach to optimizing modular vehicle fleet operation | |
| US20220263313A1 (en) | Data structure comprising an energy schedule and method for providing a data structure comprising an energy schedule | |
| CN115964182B (en) | Resource scheduling method and system | |
| CN107274135A (en) | The raw tobacco material overall planning method and system shared based on cooperative information | |
| US20140149346A1 (en) | System and method for maintaining distributed data transaction atomicity and isolation during a replication to a different target cluster |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181211 |