CN110968749A - Visual display device and method based on big data containerized dispatching center system - Google Patents

Visual display device and method based on big data containerized dispatching center system Download PDF

Info

Publication number
CN110968749A
CN110968749A CN201911333611.7A CN201911333611A CN110968749A CN 110968749 A CN110968749 A CN 110968749A CN 201911333611 A CN201911333611 A CN 201911333611A CN 110968749 A CN110968749 A CN 110968749A
Authority
CN
China
Prior art keywords
content
module
workflow
big data
node
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201911333611.7A
Other languages
Chinese (zh)
Inventor
彭锋
宋文欣
金雷
张飞
黄艳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuhan Linktime Cloud Technology Co ltd
Original Assignee
Wuhan Linktime Cloud Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuhan Linktime Cloud Technology Co ltd filed Critical Wuhan Linktime Cloud Technology Co ltd
Priority to CN201911333611.7A priority Critical patent/CN110968749A/en
Publication of CN110968749A publication Critical patent/CN110968749A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/903Querying
    • G06F16/9035Filtering based on additional data, e.g. user or group profiles
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/903Querying
    • G06F16/9038Presentation of query results
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/0486Drag-and-drop

Abstract

The invention provides a visual display device and method based on a big data containerization dispatching center system, wherein the device comprises the following steps: the workflow creating module creates at least one workflow according to business needs, manually drags each module operation in the big data processing process to the workflow, and self-defines the dependency relationship among each module operation in the workflow; the positioning module positions required content according to the input search content; the filtering module filters the content according to the positioned content; the display module displays the positioned content and the filtered content to a user in real time; and the highlight module highlights all the dependency relations related to the contents when a mouse is arranged on the displayed contents and pops up the detailed information of the current contents. The invention realizes clear, concise and telescopic panoramic display in a big data containerization dispatching center system and describes the job dependency relationship in the workflow.

Description

Visual display device and method based on big data containerized dispatching center system
Technical Field
The invention relates to the field of communication, in particular to a visual display device and method based on a big data containerization dispatching center system.
Background
The directed Acyclic Graph (Direct Acyclic Graph) supports asynchronous concurrency, can form a topological tree structure, and can greatly improve expansibility. Directed acyclic graphs are widely applied to complex data relation graph display.
The workflow of the Apache Oozie big data scheduling framework is a set of actions (Hadoop map/reduce operation, pig operation and other operations), and is controlled by a control dependency DAG (direct Acyclic graph), but the interface has no user interaction, and the establishment of the operation dependency relationship is not flexible.
The Airflow is a platform for arranging, scheduling and monitoring the workflow, the Airflow arranges the workflow into DAGs formed by tasks, a scheduler executes the tasks according to a specified dependency relationship in a group of running programs, and the execution sequence of all the tasks in the workflow and the dependency relationship among the tasks can be described. But is limited to a single workflow and does not have a panoramic description of the data relationships of all the workflows of the system.
The button is a data integration module in the whole Pentaho product system, and a user can create and define a job or conversion in a dragging mode and provide execution conditions of the job or conversion of each child node. Jobs in a workflow are executed in a serial fashion. A Job comprises one or more Job items, and the Job items are connected by a Hop (Job Hop), namely a connecting line with arrows between steps. The execution sequence of the jobs is determined by jump and the execution result of each job item, and the user can customize the operation. However, when too many jobs are performed in the workflow and the nesting relationship between the jobs and the job items is too complex, the display of the dependency relationship between the jobs and the job items becomes not friendly enough, and a clear, concise and telescopic panoramic display does not exist.
Based on this, the existing visualization interface of the big data dispatching center system mainly has the following defects: the interactive experience of the visual interface of the existing big data scheduling center system is poor; a clear, concise and telescopic panoramic display is lacked to describe the job dependency relationship in the workflow; and there is no panoramic presentation of all workflows and all jobs across the system level of the workflow.
Disclosure of Invention
In order to solve the technical problems, the invention provides a visual display device and method based on a big data containerized dispatching center system, so as to realize clear, concise and telescopic panoramic display in the big data containerized dispatching center system and describe the job dependency relationship in the workflow.
According to an embodiment of the present invention, the present invention provides a visual display device based on a big data containerized dispatch center system, the device includes:
the workflow creating module is used for creating at least one workflow according to the business requirements;
the positioning module is used for positioning needed contents according to the input search contents;
the filtering module is used for filtering the content according to the positioned content;
the display module is used for displaying the positioned content and the filtered content to a user in real time;
and the highlight module is used for highlighting when the mouse is arranged on the displayed content and popping up detailed information of the current content.
Further, the workflow creating module is configured to create at least one workflow according to the service requirement, where the creating includes adding a Sqoop job, a Hive job, a Spark job, a data conversion job, or a custom job, manually dragging each module job to the workflow in the big data processing process, and custom-defining a dependency relationship between each module of the workflow.
Further, the filtered content includes the time consumed by the job and the CPU occupation ratio.
Further, the display module is used for displaying the positioned content and the filtered content to a user in real time, wherein the displaying includes displaying the directed acyclic graph DAG in a force-directed graph mode so as to clearly display a relation panorama among all data sources and operations of the system.
Further, the highlighting module is used for highlighting when the mouse is arranged on the displayed content and popping up detailed information of the current content, and comprises the steps of placing the mouse on a content node, highlighting upstream and downstream dependent nodes and clicking the highlighted node to pop up the detailed information.
Further, the detailed information includes specific information of the node where the content is located, node dependency information, and all upstream and downstream dependency details of the node.
Further, the displaying module is configured to display the located content and the filtered content to the user in real time, where the displaying includes displaying in a manner of a star-shaped graph or a single directed acyclic graph DAG.
Further, the shape, the size and the dimension of the operation presentation are set in a custom mode.
Furthermore, the display module also includes setting the node configuration, connection color, size and shape of the content when displaying the content.
According to an embodiment of the present invention, the present invention further provides a visualization display method based on a big data containerization scheduling center system, where the method includes:
s1, creating one or more workflows based on business requirements, wherein the workflows comprise Sqoop adding operation, Hive operation, Spark operation, data conversion operation or custom operation, and manually dragging to establish relationship dependence;
s2, inputting contents to be searched, positioning the contents, and displaying the directed acyclic graph in a force-directed graph mode to clearly display a panoramic graph of the relationship between all data sources and the operation of the system;
s3, filtering out contents meeting conditions, wherein the filtered contents comprise operation time consumption and a CPU occupation ratio;
s4, placing a mouse on a node of the content through all workflow contents and upstream and downstream dependency relations in the directed acyclic graph panorama display system, and enabling upstream and downstream dependency nodes to be highlighted;
s5, clicking the highlight node to pop up detailed information of the node, wherein the detailed information comprises specific information, node dependence information and all upstream and downstream dependence details of the node.
The invention provides a visual display device and method based on a big data containerization dispatching center system, which have the following beneficial effects:
(1) the method supports the manual dragging and adding of the dependency relationship of the data processing operation of the big data containerized dispatching center system by the user;
(2) the method takes a data source as a center, and radiates Sqoop operation, Hive operation, Spark operation, data conversion operation, custom operation and the like of a big data platform around the data source;
(3) according to the method, the operation/data source is clicked, the upstream and downstream dependency relationship can be observed, and the operation can be filtered according to the condition;
(4) the invention supports star graphs and single DAG graphs, configuration nodes, connection line colors, shapes, sizes and the like, positioning, zooming and checking the related connection of single nodes.
Drawings
Fig. 1 is a frame diagram of a visual display device based on a big data containerization dispatching center system provided by the invention;
fig. 2 is a flowchart of a visual display method based on a big data containerization scheduling center system according to the present invention.
Detailed Description
For the convenience of understanding, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As shown in fig. 1, the visual display device based on the big data containerized dispatch center system provided by the present invention includes:
the workflow creating module is used for creating at least one workflow according to business requirements, creating the workflow, adding Sqoop operation, Hive operation, Spark operation, data conversion operation or custom operation, manually dragging each module operation in the big data processing process to the workflow, and custom-defining the dependency relationship among each module of the workflow; according to the invention, a plurality of workflows can be created according to requirements; as jobs and workflows increase, clarifying relationships between data sources and jobs becomes increasingly important.
The display device is provided with a search box, so that the required content is input through the search box of the instrument panel, and the specific node based on the content is further positioned.
The filtering module is used for filtering the content according to the positioned content, and in the invention, the content is filtered through the filtering module of the instrument panel, so that content nodes meeting conditions, such as time consumption of operation, CPU occupation proportion and the like, are filtered.
And the display module is used for displaying the positioned content and the filtered content to a user in real time. In the invention, the display module plays the positioned content and the filtered content, and the specific display mode comprises displaying the directed acyclic graph DAG in a force-directed graph mode so as to clearly display the relation panoramic graph between all data sources and the operation of the system. Of course, the display mode also includes a mode of a star map or a single DAG map for displaying, and meanwhile, in the invention, the display modes of various jobs can customize the contents of the display shape, size and the like of the jobs. Meanwhile, the nodes of the required content can be configured based on the workflow in the display module, and the colors, the sizes and the shapes of the connecting lines can be set, so that personalized display is realized.
And the highlight module is used for highlighting when the mouse is arranged on the displayed content and popping up detailed information of the current content. In the embodiment of the invention, the highlight module realizes corresponding functions based on the cooperation of a mouse, specifically, the mouse is placed on a content node, the upstream and downstream dependent nodes are highlighted, and detailed information is popped up by clicking the highlighted node. The detailed information comprises specific information of a node where the content is located, node dependency information and all upstream and downstream dependency details of the node.
According to another embodiment of the present invention, as shown in fig. 2, the present invention further provides a visualization display method based on a big data containerization dispatch center system, where the method includes:
s1, creating one or more workflows based on business requirements, wherein the workflows comprise Sqoop adding operation, Hive operation, Spark operation, data conversion operation or custom operation, and manually dragging to establish relationship dependence;
s2, inputting contents to be searched, positioning the contents, and displaying the directed acyclic graph in a force-directed graph mode to clearly display a panoramic graph of the relationship between all data sources and the operation of the system;
s3, filtering out the content meeting the conditions and displaying the content, wherein the filtered content comprises the time consumed by the operation and the proportion occupied by a CPU;
s4, placing a mouse on a node of the content through all workflow contents and upstream and downstream dependency relations in the directed acyclic graph panorama display system, and enabling upstream and downstream dependency nodes to be highlighted;
s5, clicking the highlight node to pop up detailed information of the node, wherein the detailed information comprises specific information, node dependence information and all upstream and downstream dependence details of the node.
The operation assembly line provided by the invention supports the interactive experience of users on the manual free dragging of the dependence relationship of the large data containerization data processing operation and the data source. Meanwhile, the task panorama of the invention displays the chart by receiving a specific data structure. Icons support two types: a. the star-shaped graph supports two forms of a center form and a centerless form, and a center point can be configured; b. and the single DAG graph displays the upstream and downstream relations of the single node and displays the details of the node. Node and connecting line design among the icons supports node color configuration, style when a hover mouse is suspended, highlighted style, shape, size and the like. The graph operation supports the enlargement and the reduction, the node aggregation, the highlight and the node connection behaviors are controlled by clicking self-definition, and the number and the positioning of the initial nodes of the window are controlled.
It will be evident to those skilled in the art that the embodiments of the present invention are not limited to the details of the foregoing illustrative embodiments, and that the embodiments of the present invention are capable of being embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the embodiments being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. Several units, modules or means recited in the system, apparatus or terminal claims may also be implemented by one and the same unit, module or means in software or hardware. The terms first, second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only used for illustrating the technical solutions of the embodiments of the present invention and not for limiting, and although the embodiments of the present invention are described in detail with reference to the above preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions can be made on the technical solutions of the embodiments of the present invention without departing from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. A visual display device based on a big data containerized dispatching center system is characterized by comprising:
the workflow creating module is used for creating at least one workflow according to the business requirements;
the positioning module is used for positioning needed contents according to the input search contents;
the filtering module is used for filtering the content according to the positioned content;
the display module is used for displaying the positioned content and the filtered content to a user in real time;
and the highlight module is used for highlighting when the mouse is arranged on the displayed content and popping up detailed information of the current content.
2. The visual display device based on the big data containerization dispatching center system according to claim 1, wherein the workflow creating module is configured to create at least one workflow according to business requirements, and includes adding an Sqoop job, a Hive job, a Spark job, a data conversion job, or a custom job, manually dragging each module job to the workflow in the big data processing process, and custom-defining a dependency relationship between each module of the workflow.
3. The visual display device of the big data containerization-based dispatch center system of claim 1, wherein the filtered content comprises job time consumption and CPU occupation ratio.
4. The big-data containerization scheduling center system-based visual display device of claim 1, wherein the display module is configured to display the positioned content and the filtered content to the user in real time, and the display module is configured to display the directed acyclic graph DAG in a force-directed graph manner to clearly display a panoramic view of relationships between all data sources and jobs of the system.
5. The visual display device of the big data containerization based dispatch center system of claim 1, wherein the highlighting module is configured to highlight when a mouse is placed on the displayed content and pop up the detailed information of the current content, and the highlighting module is configured to place the mouse on the content node, highlight the dependent nodes upstream and downstream, and pop up the detailed information by clicking the highlighted node.
6. The big data containerization based dispatch center system visual display of claim 1, wherein the detailed information comprises specific information of a node where the content is located, node dependency information, and all upstream and downstream dependency details of the node.
7. The visual display device of the big-data containerized scheduling center system according to claim 4, wherein the display module for displaying the located content and the filtered content to the user in real time further comprises displaying by means of a star-shaped graph or a single directed acyclic graph DAG.
8. The big data containerization based dispatch center system visual display of claim 2, wherein the job presentation shape, size, dimensions are set in a custom manner.
9. The visual display device of the big data containerization-based dispatch center system of claim 7, wherein the display module further comprises a configuration, a connection color, a size and a shape of the nodes of the content when displaying the content.
10. A visual display method based on a big data containerization dispatching center system is characterized by comprising the following steps:
s1, creating one or more workflows based on business requirements, wherein the workflows comprise Sqoop adding operation, Hive operation, Spark operation, data conversion operation or custom operation, and manually dragging to establish relationship dependence;
s2, inputting contents to be searched, positioning the contents, and displaying the directed acyclic graph in a force-directed graph mode to clearly display a panoramic graph of the relationship between all data sources and the operation of the system;
s3, filtering out contents meeting conditions, wherein the filtered contents comprise operation time consumption and a CPU occupation ratio;
s4, placing a mouse on a node of the content through all workflow contents and upstream and downstream dependency relations in the directed acyclic graph panorama display system, and enabling upstream and downstream dependency nodes to be highlighted;
s5, clicking the highlight node to pop up detailed information of the node, wherein the detailed information comprises specific information, node dependence information and all upstream and downstream dependence details of the node.
CN201911333611.7A 2019-12-23 2019-12-23 Visual display device and method based on big data containerized dispatching center system Pending CN110968749A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911333611.7A CN110968749A (en) 2019-12-23 2019-12-23 Visual display device and method based on big data containerized dispatching center system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911333611.7A CN110968749A (en) 2019-12-23 2019-12-23 Visual display device and method based on big data containerized dispatching center system

Publications (1)

Publication Number Publication Date
CN110968749A true CN110968749A (en) 2020-04-07

Family

ID=70035914

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911333611.7A Pending CN110968749A (en) 2019-12-23 2019-12-23 Visual display device and method based on big data containerized dispatching center system

Country Status (1)

Country Link
CN (1) CN110968749A (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111651460A (en) * 2020-06-11 2020-09-11 上海德易车信息科技有限公司 Data management method and device, electronic equipment and readable storage medium
CN112817744A (en) * 2021-01-14 2021-05-18 内蒙古蒙商消费金融股份有限公司 Task processing method and device
CN112836323A (en) * 2021-01-08 2021-05-25 广东电网有限责任公司广州供电局 Station line user-changing-based equipment map and search visualization method

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160358101A1 (en) * 2015-06-05 2016-12-08 Facebook, Inc. Machine learning system interface
CN106233261A (en) * 2014-04-17 2016-12-14 起元技术有限责任公司 The integration of processing environment monitors and controls
CN106293928A (en) * 2015-06-05 2017-01-04 阿里巴巴集团控股有限公司 A kind of overall situation task node dependence method for visualizing, device and system
US20170154088A1 (en) * 2015-11-30 2017-06-01 Tableau Software, Inc. Systems and Methods for Interactive Visual Analysis Using A Specialized Virtual Machine
CN108460083A (en) * 2018-01-16 2018-08-28 浙江大学 A kind of knowledge mapping visual query tool
CN109800081A (en) * 2018-12-14 2019-05-24 深圳壹账通智能科技有限公司 A kind of management method and relevant device of big data task
CN110019492A (en) * 2017-09-14 2019-07-16 阿里巴巴集团控股有限公司 A kind of information display method and device

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106233261A (en) * 2014-04-17 2016-12-14 起元技术有限责任公司 The integration of processing environment monitors and controls
US20160358101A1 (en) * 2015-06-05 2016-12-08 Facebook, Inc. Machine learning system interface
CN106293928A (en) * 2015-06-05 2017-01-04 阿里巴巴集团控股有限公司 A kind of overall situation task node dependence method for visualizing, device and system
US20170154088A1 (en) * 2015-11-30 2017-06-01 Tableau Software, Inc. Systems and Methods for Interactive Visual Analysis Using A Specialized Virtual Machine
CN110019492A (en) * 2017-09-14 2019-07-16 阿里巴巴集团控股有限公司 A kind of information display method and device
CN108460083A (en) * 2018-01-16 2018-08-28 浙江大学 A kind of knowledge mapping visual query tool
CN109800081A (en) * 2018-12-14 2019-05-24 深圳壹账通智能科技有限公司 A kind of management method and relevant device of big data task

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111651460A (en) * 2020-06-11 2020-09-11 上海德易车信息科技有限公司 Data management method and device, electronic equipment and readable storage medium
CN112836323A (en) * 2021-01-08 2021-05-25 广东电网有限责任公司广州供电局 Station line user-changing-based equipment map and search visualization method
CN112817744A (en) * 2021-01-14 2021-05-18 内蒙古蒙商消费金融股份有限公司 Task processing method and device

Similar Documents

Publication Publication Date Title
CN110968749A (en) Visual display device and method based on big data containerized dispatching center system
CN105608319B (en) A kind of mask method and annotation equipment of digital pathological section
CN107818479A (en) Favor information method for pushing, system, storage medium and electronic equipment
CN113031946B (en) Method and device for rendering page component
US9176727B2 (en) Infrastructure software patch reporting and analytics
CN108369514B (en) System and method for executable content and executable content stream creation
WO2024088207A1 (en) View generation method and apparatus, electronic device, and storage medium
WO2020004156A1 (en) Display control device, display control method, and display control program
CN110020376A (en) Data linkage method and system
WO2021098579A1 (en) Chart drawing method, device, apparatus and storage medium
WO2023222043A1 (en) View management method and apparatus, electronic device, and storage medium
CN108460818A (en) A kind of method and device of data export and animation play
CN107333145B (en) A kind of virtual gift display method and device
CN114356761A (en) Test case document generation method and device, storage medium and computer equipment
WO2023045412A1 (en) Page creation method, apparatus and system, computer device and storage medium
US20220326901A1 (en) Data display
CN115617441A (en) Method and device for binding model and primitive, storage medium and computer equipment
US20180158005A1 (en) Computer implemented methods and systems for managing tasks
CN115130442A (en) Report generation method and device, storage medium and computer equipment
CN114995931A (en) Display method and device of project bulletin board, storage medium and computer equipment
EP4327260A1 (en) Dynamic graphical containers
US11487411B2 (en) Context-driven group pill in a user interface
CN114489910B (en) Video conference data display method, device, equipment and medium
CN109587038A (en) Information push method and device
CN109190992A (en) A kind of generation method and device of the operation flow based on use-case

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20200407

RJ01 Rejection of invention patent application after publication