CN112579675A - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN112579675A CN112579675A CN201910933143.0A CN201910933143A CN112579675A CN 112579675 A CN112579675 A CN 112579675A CN 201910933143 A CN201910933143 A CN 201910933143A CN 112579675 A CN112579675 A CN 112579675A
- Authority
- CN
- China
- Prior art keywords
- data
- format
- distributed
- acquisition component
- computing system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/565—Conversion or adaptation of application format or content
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computing Systems (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention provides a data processing method and a device, wherein the method comprises the following steps: determining a protocol of any hardware device in the operation and maintenance technology network for transmitting data; determining target acquisition components from at least one preset acquisition component, wherein each acquisition component can analyze a protocol used for transmitting data, and each target acquisition component is an acquisition component capable of analyzing the protocol used for transmitting data of the hardware equipment; calling a target acquisition component to acquire data output by the hardware equipment, analyzing the data output by the hardware equipment, and generating analyzed data; converting the analyzed data into data in a distributed format, wherein the distributed format is a data format required by a distributed computing system; and outputting the data in the distributed format to the distributed computing system. The invention provides a data processing method and device, which can more conveniently generate data in a data format required by a distributed computing system.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data processing method and apparatus.
Background
The intelligent manufacturing needs to collect data output by hardware devices of an operation and maintenance Technology (OT) network and analyze characteristics of the data, and on the basis, an intelligent environment with interconnected, mutual and distributed nodes is created. The failure and the prediction demand are predicted by carrying out distributed calculation and intelligent analysis on the data output by the hardware equipment of the operation and maintenance technology network, a new value point in the manufacturing process is found, and the problem of unforeseen management is solved, so that the data output by the hardware equipment of the operation and maintenance technology network has a higher value.
At present, hardware devices such as numerous intelligent sensing terminals are deployed in industrial production, and data are continuously generated and output. Distributed computing systems cannot directly use this data. In order to provide usable data to a distributed computing system when using the distributed computing system, developers need to face heterogeneous hardware environments, analyze protocols for transmitting data of respective hardware devices, write data collection codes, and deploy operating environments, which requires a lot of effort and time, and is very complicated.
Disclosure of Invention
The embodiment of the invention provides a data processing method and device, which can be used for more conveniently generating data in a data format required by a distributed computing system.
In a first aspect, an embodiment of the present invention provides a data processing method, including:
determining a protocol for transmitting data of any hardware device in an operation and maintenance technology network, wherein the operation and maintenance technology network is an industrial communication network for connecting devices and systems in a production field;
determining target acquisition components from at least one preset acquisition component, wherein each acquisition component can analyze a protocol used for transmitting data, and the target acquisition components can analyze the protocol used for transmitting data of the hardware equipment;
calling the target acquisition component to acquire data output by the hardware equipment, analyzing the data output by the hardware equipment, and generating analyzed data;
converting the analyzed data into data in a distributed format, wherein the distributed format is a data format required by a distributed computing system;
and outputting the data in the distributed format to the distributed computing system.
Alternatively,
the converting the parsed data into data in a data format required by a computing system during distribution includes:
converting the analyzed data into data in a preset standard format;
and converting the data in the standard format into the data in the distributed format.
Alternatively,
after converting the parsed data into the data in the preset standard format and before converting the parsed data into the data in the distributed format, the method further includes:
caching the data in the standard format into a buffer pool;
and acquiring the data in the standard format from the buffer pool.
Alternatively,
the method further comprises the following steps:
acquiring the data in the standard format from the buffer pool;
converting the data in the standard format into data in a database format;
storing the data in the database format into a database;
when a data request sent by the distributed computing system is received, judging whether data requested by the data request exists in the buffer pool, and if the data requested by the data request does not exist in the buffer pool, acquiring data in a database format requested by the data request from the database;
converting the data in the database format requested by the data request into the data in the distributed format, and outputting the data in the distributed format to the distributed computing system.
Alternatively,
each of the acquisition components includes: an interface for a request mode and an interface for a push mode;
before the invoking the target obtaining component to obtain the data output by the hardware device, further comprising:
determining a mode of the hardware device for outputting data;
when the mode of the hardware equipment for outputting data is a request mode, an interface used for requesting the mode in the target acquisition component is realized;
and when the mode of the hardware equipment for outputting data is a push mode, realizing an interface used for the push mode in the target acquisition component.
Alternatively,
the converting the data in the standard format into the data in the distributed format includes:
converting the data in the standard format into data in a kafka data format by using a kafka program;
wherein the distributed format comprises: the kafka data format;
the outputting the data in the distributed format to the distributed computing system includes:
inputting the data in the kafka data format into a queue to enable the distributed computing system to obtain the data in the kafka data format from the queue.
Alternatively,
the method further comprises the following steps:
receiving an externally input command format of an issued message and a command format of a data return value of the acquisition component;
and generating the acquisition component according to the command format of the sent message and the command format of the data return value.
Alternatively,
the data in the standard format comprises: any one or more of a name of the data, a value of the data, a time at which the data is generated, a type of the value of the data, a unit of the value of the data, and a state of a hardware device that generates the data.
In a second aspect, an embodiment of the present invention provides a device for processing tool data, including:
at least one acquisition component, an adapter module and a conversion module;
each acquisition component is capable of parsing a protocol for transmitting data;
the adapter module is used for determining a protocol for transmitting data of any hardware device in an operation and maintenance technical network, determining a target acquisition component from at least one preset acquisition component, and calling the target acquisition component, wherein the operation and maintenance technical network is an industrial communication network for connecting equipment and a system on a production field, and the target acquisition component is an acquisition component capable of analyzing the protocol for transmitting data of the hardware device;
the acquisition component is used for acquiring the data output by the hardware equipment, analyzing the data output by the hardware equipment and generating analyzed data;
the conversion module is used for converting the analyzed data into data in a distributed format and outputting the data in the distributed format to the distributed computing system, wherein the distributed format is a data format required by the distributed computing system.
Alternatively,
the conversion module is used for converting the analyzed data into data in a preset standard format and converting the data in the standard format into the data in the distributed format.
Alternatively,
the conversion module is further configured to, after converting the parsed data into the preset data in the standard format and before converting the parsed data into the data in the distributed format, cache the data in the standard format in a buffer pool, and acquire the data in the standard format from the buffer pool.
Alternatively,
the apparatus further comprises: a data persistence module;
the data persistence module is used for acquiring the data in the standard format from the buffer pool, converting the data in the standard format into the data in the database format, and storing the data in the database format into a database;
the conversion module is further configured to, when a data request sent by the distributed computing system is received, determine whether data requested by the data request exists in the buffer pool, if not, obtain data in a database format requested by the data request from the database, convert the data in the database format requested by the data request into the data in the distributed format, and output the data in the distributed format to the distributed computing system.
Alternatively,
each of the acquisition components includes: an interface for a request mode and an interface for a push mode;
the adapter module is further configured to determine a mode in which the hardware device outputs data before the target acquisition component is called to acquire the data output by the hardware device, implement an interface for requesting the mode in the target acquisition component when the mode in which the hardware device outputs data is a request mode, and implement an interface for pushing the mode in the target acquisition component when the mode in which the hardware device outputs data is a push mode.
Alternatively,
the conversion module is used for converting the data in the standard format into the data in the kafka data format by using a kafka program, and inputting the data in the kafka data format into a queue so that the distributed computing system acquires the data in the kafka data format from the queue;
wherein the distributed format comprises: the kafka data format.
Alternatively,
the apparatus further comprises: a component generation module;
the component generating module is used for receiving an externally input command format of an issued message and a command format of a data return value of the acquiring component, and generating the acquiring component according to the command format of the issued message and the command format of the data return value.
Alternatively,
the data in the standard format comprises: any one or more of a name of the data, a value of the data, a time at which the data is generated, a type of the value of the data, a unit of the value of the data, and a state of a hardware device that generates the data.
In the embodiment of the invention, each acquisition component can analyze a protocol for transmitting data, and the acquisition component is used for acquiring data output by corresponding hardware equipment, wherein the acquisition component can analyze the protocol for transmitting data of the corresponding hardware equipment, analyze the acquired data, and convert the analyzed data into data in a data format required by the distributed computing system.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention;
FIG. 2 is a flow chart of another data processing method according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention.
Description of reference numerals:
301 acquisition component 302 adapter module 303 conversion module
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer and more complete, the technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention, and based on the embodiments of the present invention, all other embodiments obtained by a person of ordinary skill in the art without creative efforts belong to the scope of the present invention.
As shown in fig. 1, an embodiment of the present invention provides a data processing method, which may include the following steps:
step 102: determining target acquisition components from at least one preset acquisition component, wherein each acquisition component can analyze a protocol used for transmitting data, and the target acquisition components can analyze the protocol used for transmitting data of the hardware equipment;
step 103: calling the target acquisition component to acquire data output by the hardware equipment, analyzing the data output by the hardware equipment, and generating analyzed data;
step 104: converting the analyzed data into data in a distributed format, wherein the distributed format is a data format required by a distributed computing system;
step 105: and outputting the data in the distributed format to the distributed computing system.
In the embodiment of the invention, each acquisition component can analyze a protocol for transmitting data, and the acquisition component is used for acquiring data output by corresponding hardware equipment, wherein the acquisition component can analyze the protocol for transmitting data of the corresponding hardware equipment, analyze the acquired data, and convert the analyzed data into data in a data format required by the distributed computing system.
In the embodiment of the invention, corresponding acquisition components are respectively arranged aiming at different protocols for transmitting data, so that the method and the device can be suitable for various different protocols, and a user only needs to be directly connected with hardware equipment in an operation and maintenance technical network through the acquisition components. Wherein, the hardware equipment in the operation and maintenance technology network includes: various sensors, PLC (Programmable Logic Controller), OPC UA (OLE for Process Control Unified Architecture), servers, etc., which are used for transmitting data, and include: OPC UA protocol, PROFIBUS protocol, ModBus protocol, etc. The acquisition component can be directly connected with each sensor to acquire the data sensed by the sensor. The acquisition component can be connected with the PLC, the PLC acquires data output by hardware equipment such as various connected sensors, and then the acquisition component acquires the data from the PLC, and can also acquire data generated by the PLC from the PLC. The acquisition component can be connected with the OPC UA server, the OPC UA server acquires data output by the connected numerical control devices, the acquisition component acquires data output by the numerical control devices from the OPC UA server, and the acquisition component can also acquire data generated by the OPC UA server from the OPC UA server.
In an embodiment of the present invention, the converting the parsed data into data in a data format required by a computing system during distribution includes:
converting the analyzed data into data in a preset standard format;
and converting the data in the standard format into the data in the distributed format.
In the embodiment of the invention, after the data in the standard format is converted, when the data output by the hardware equipment is processed, the data in the standard format can be processed without processing the data in various different protocols, so that the data can be processed more conveniently.
In an embodiment of the present invention, the data in the standard format includes: any one or more of a name of the data, a value of the data, a time at which the data is generated, a type of the value of the data, a unit of the value of the data, and a state of a hardware device that generates the data.
Specifically, the standard format may be as follows:
{ "name": "xx", "status": "xx", "value": "xx", "time": "xx", "value type": "xx", "unit": "xx" }
Where "name" refers to the name of the data, "state" refers to the state of the hardware device that generated the data, "value" refers to the value of the data, "time" refers to the time at which the data was generated, "value type" refers to the type of value of the data, and "unit" refers to the unit of value of the data.
Taking data output by a temperature sensor as an example, the following data in the standard format is obtained after the data is converted into data in the standard format:
{ "name": "temperature", "state": "normal", "value": "38.22", "time": "2019-04-20", "value type": "numric", "unit": "° c".
Wherein, each item in the standard format can be adjusted according to the need, for example: the term "status" may not be included.
In an embodiment of the present invention, after converting the parsed data into the data in the preset standard format, before converting the parsed data into the data in the distributed format, the method further includes:
caching the data in the standard format into a buffer pool;
and acquiring the data in the standard format from the buffer pool.
In the embodiment of the invention, because the buffer pool is only used for buffering the data output by the hardware device, the buffer pool does not retain excessive data, only retains the latest data, and the new data can overwrite the old data, thereby ensuring that the buffer pool does not overflow and ensuring that a consumer program needing the data can obtain the latest data. In addition, when the data in the buffer pool is updated, the consumer can be informed that the program data is updated.
Wherein, the buffer pool (buffer pool) is composed of a plurality of buffer areas with the same size. When a process needs to use a buffer, an application is made, the application is distributed to the process by a management program, and the buffer is released after the application is finished, so that a small amount of buffer can be used for serving more processes.
The buffer is a part of the memory space. That is, a certain amount of storage space is reserved in the memory space for buffering the input or output data, and the reserved space is called a buffer, and the buffer has a certain size.
In an embodiment of the invention, the method further comprises:
acquiring the data in the standard format from the buffer pool;
converting the data in the standard format into data in a database format;
storing the data in the database format into a database;
when a data request sent by the distributed computing system is received, judging whether data requested by the data request exists in the buffer pool, and if the data requested by the data request does not exist in the buffer pool, acquiring data in a database format requested by the data request from the database;
converting the data in the database format requested by the data request into the data in the distributed format, and outputting the data in the distributed format to the distributed computing system.
In the embodiment of the invention, the data output by the hardware device is persistently stored in the database, and when the data requested by the data request of the distributed computing system is not in the buffer pool, the data requested by the data request can be obtained from the database. Since the distributed computing system cannot directly use the data in the database format, it is necessary to convert the data in the database format into the data in the distributed format.
In the embodiment of the invention, when a plurality of different consumer programs acquire data from the buffer pool, an independent thread can be set for each consumer program, so that the plurality of consumer programs can run simultaneously, the consumer programs cannot influence each other, and different requirements of different consumer programs on data processing efficiency and data timeliness can be met.
For example, there are three consumer programs:
the first consumer program is a distributed format conversion program which is used for converting data in a standard format into data in a distributed format;
the second consumer program is a conversion program in a database format, and the conversion program in the database format is used for converting data in a standard format into data in the database format;
the third consumer program is a cloud-formatted converter program for converting data in a standard format into data suitable for storage in the cloud.
In an embodiment of the invention, the method further comprises:
receiving an externally input command format of an issued message and a command format of a data return value of the acquisition component;
and generating the acquisition component according to the command format of the sent message and the command format of the data return value.
In the embodiment of the present invention, each acquisition component may be generated in this manner, and in addition, a user may also define an acquisition component by using the implementation manner provided in the embodiment of the present invention. If there is no preset acquisition component capable of analyzing the current protocol, then the command format of the sent message and the command format of the data return value may be determined according to the current protocol, and an acquisition component capable of analyzing the current protocol is generated.
In an embodiment of the present invention, each of the obtaining components includes: an interface for a request mode and an interface for a push mode;
before the invoking the target obtaining component to obtain the data output by the hardware device, further comprising:
determining a mode of the hardware device for outputting data;
when the mode of the hardware equipment for outputting data is a request mode, an interface used for requesting the mode in the target acquisition component is realized;
and when the mode of the hardware equipment for outputting data is a push mode, realizing an interface used for the push mode in the target acquisition component.
In the embodiment of the present invention, when determining the mode in which the hardware device outputs data, the determination may be based on initialization information of the hardware device. The interfaces of two modes are arranged in each acquisition assembly, so that different hardware equipment can be connected more conveniently.
In particular, all acquisition components inherit an acquisition interface that includes the following member functions: initializing a function, sending a requested function, receiving data function, subscribing function, and monitoring data function. The function of sending the request and the function of receiving the data are used for realizing an interface for a request mode, and the function of subscribing and the function of monitoring the data are used for realizing an interface for a push mode.
And when the mode of the hardware equipment for outputting data is a request mode, realizing a function for sending a request and a function for receiving data in the current acquisition component so as to realize an interface for requesting the mode.
And when the mode of the hardware equipment for outputting the data is a push mode, realizing a function of subscribing and a function of monitoring the data in the current acquisition assembly so as to realize an interface for the push mode.
In an embodiment of the present invention, the converting the data in the standard format into the data in the distributed format includes:
converting the data in the standard format into data in a kafka data format by using a kafka program;
wherein the distributed format comprises: the kafka data format;
the outputting the data in the distributed format to the distributed computing system includes:
inputting the data in the kafka data format into a queue to enable the distributed computing system to obtain the data in the kafka data format from the queue.
In the embodiment of the present invention, the conversion from the standard format to the distributed format may be realized by the kafka program. The converted data in the kafka data format is stored in a queue, and the distributed computing system can automatically acquire the data in the corresponding kafka data format from the queue.
Wherein the queue is a special linear table, the queue allows deletion operations at the front end (front) of the table and insertion operations at the back end (rear) of the table, and like the stack, the queue is a linear table with restricted operations. The end performing the insert operation is called the tail of the queue, and the end performing the delete operation is called the head of the queue.
As shown in fig. 2, an embodiment of the present invention provides a data processing method, which may include the following steps:
step 201: a protocol for transmitting data of any hardware device in the operation and maintenance technology network is determined.
The operation and maintenance technology network is an industrial communication network used for connecting equipment and a system on a production field.
Specifically, the protocol of any hardware device in the operation and maintenance technology network for transmitting data may be determined according to externally input configuration information of the hardware device. The configuration information may include: the name of the hardware device, the address of the hardware device, the type of protocol of the hardware device used for transmitting data, the data type of the data output by the hardware device, the unit (such as DEG C, meter, kilogram and the like) of the data output by the hardware device, and the related parameters required by the protocol of the hardware device used for transmitting data.
Before invoking the target acquisition component, the target acquisition component may be initialized according to configuration information of the hardware device. Specifically, the address of the hardware device and the like are configured.
For example, the hardware device is a PLC, and the protocol of the PLC for transmitting data is determined to be modbus protocol.
Step 202: and determining a target acquisition component from at least one preset acquisition component.
Each acquisition component can analyze a protocol used for transmitting data, and the target acquisition component is an acquisition component capable of analyzing the protocol used for transmitting data of the hardware equipment.
For example, an acquisition component is set for the ModBus protocol, and for the PLC, an acquisition component capable of resolving the ModBus protocol is determined.
Step 203: and calling the target acquisition component to acquire the data output by the hardware equipment, analyzing the data output by the hardware equipment, and generating the analyzed data.
For example, the information frame transmitted in the ModBus protocol includes: address code, function code, data area and error check code. The acquisition component capable of analyzing the ModBus protocol can analyze the content of the data area according to the structure of the information frame, and then can be standardized according to the content of the data area when standardization is carried out.
Step 204: and converting the analyzed data into data in a preset standard format.
Specifically, the standard format may be:
{ "name": "xx", "status": "xx", "value": "xx", "time": "xx", "value type": "xx", "unit": "xx" }
For data output by hardware devices of the operation and maintenance technology network, the standard format can contain all information of data output by various hardware devices.
The data in the standard format can be simple text information, and all consumer programs needing to acquire the data in the standard format from the buffer pool can be identified and analyzed, so that subsequent processing processes such as format conversion and the like are conveniently performed.
Step 205: and caching the data in the standard format into a buffer pool.
Specifically, the latest data can be held by the buffer pool. The buffer pool can be used for facilitating a plurality of different consumer programs to obtain the latest data.
Step 206: and acquiring the data in the standard format from the buffer pool.
Step 207: the data in the standard format is converted into data in the kafka data format using the kafka program.
Specifically, data in the standard format may be converted into data in the kafka data format by the kafka program. The kafka program may seamlessly interface with the distributed computing system, however, the kafka program may not be able to parse data output by various hardware devices in the operation and maintenance technology network. The standardized data in the standard format can be identified by the kafka program, and subsequent format conversion can be performed.
Step 208: inputting the data in the kafka data format into a queue to enable the distributed computing system to obtain the data in the kafka data format from the queue.
In particular, the kafka program may write data in the kafka data format to the queue, such that the connected distributed computing system retrieves the required data in the kafka data format from the queue.
The Kafka program is a high throughput, distributed, publication-based messaging system that can be used for offline and online messaging.
The distributed computing system includes: hadoop, Druid, Ambari, Spark, Storm, and the like. These distributed computing systems may all use the data provided by the kafka program.
The embodiment of the invention solves the problem of data source between the data acquisition equipment and the distributed computing system in the operation and maintenance technology network, so that the distributed computing system does not need to process the bottom hardware structure and network conditions, and can directly acquire the data in the data format required by the distributed computing system.
As shown in fig. 3, an embodiment of the present invention provides a device for processing tool data, including:
at least one acquisition component 301, an adapter module 302, and a translation module 303;
each of the acquisition components 301 is capable of parsing a protocol for transmitting data;
the adapter module 302 is configured to determine a protocol for transmitting data of any hardware device in an operation and maintenance technology network, determine a target acquisition component from at least one preset acquisition component, and call the target acquisition component, where the operation and maintenance technology network is an industrial communication network for connecting a device and a system in a production field, and the target acquisition component is an acquisition component capable of analyzing the protocol for transmitting data of the hardware device;
the obtaining component 302 is configured to obtain data output by the hardware device, analyze the data output by the hardware device, and generate analyzed data;
the conversion module 303 is configured to convert the parsed data into data in a distributed format, and output the data in the distributed format to the distributed computing system, where the distributed format is a data format required by the distributed computing system.
In an embodiment of the present invention, the converting module 303 is configured to convert the analyzed data into data in a preset standard format, and convert the data in the standard format into the data in the distributed format.
In an embodiment of the present invention, the converting module 303 is further configured to, after the converting the parsed data into the preset data in the standard format, before the converting the parsed data into the data in the distributed format, cache the data in the standard format in a buffer pool, and acquire the data in the standard format from the buffer pool.
In an embodiment of the present invention, the data processing method further includes: a data persistence module;
the data persistence module is used for acquiring the data in the standard format from the buffer pool, converting the data in the standard format into the data in the database format, and storing the data in the database format into a database;
the conversion module 303 is further configured to, when a data request sent by the distributed computing system is received, determine whether data requested by the data request exists in the buffer pool, if the data requested by the data request does not exist, obtain data in a database format requested by the data request from the database, convert the data in the database format requested by the data request into the data in the distributed format, and output the data in the distributed format to the distributed computing system.
In an embodiment of the present invention, each of the obtaining components 301 includes: an interface for a request mode and an interface for a push mode;
the adapter module 302 is further configured to determine a mode of the data output by the hardware device before the target obtaining component is called to obtain the data output by the hardware device, implement an interface used for the request mode in the target obtaining component when the mode of the data output by the hardware device is the request mode, and implement an interface used for the push mode in the target obtaining component when the mode of the data output by the hardware device is the push mode.
In an embodiment of the present invention, the converting module 303 is configured to convert the data in the standard format into data in a kafka data format by using a kafka program, and input the data in the kafka data format into a queue, so that the distributed computing system obtains the data in the kafka data format from the queue;
wherein the distributed format comprises: the kafka data format.
In an embodiment of the present invention, the data processing method further includes: a component generation module;
the component generating module is used for receiving an externally input command format of an issued message and a command format of a data return value of the acquiring component, and generating the acquiring component according to the command format of the issued message and the command format of the data return value.
In an embodiment of the present invention, the data in the standard format includes: any one or more of a name of the data, a value of the data, a time at which the data is generated, a type of the value of the data, a unit of the value of the data, and a state of a hardware device that generates the data.
The data processing device provided by the embodiment of the invention can provide a visual configuration interface, and the information of hardware equipment of the operation and maintenance technology network and the like can be configured externally through the visual configuration interface.
The data processing device provided by the embodiment of the invention is suitable for industrial scenes needing distributed computing tasks and big data analysis, and can be used for processing data of a whole factory and data of a single production chain.
To avoid production downtime due to interaction between the operation and maintenance technology network and the office network, the two networks are typically separated by a firewall. The operation and maintenance technology network is a network in which production machines, controllers and various data acquisition devices operate, and the office network is a network in which upper-layer application programs operate. Distributed computing systems are typically deployed on office networks.
In the embodiment of the present invention, the processing apparatus for data provided in the embodiment of the present invention may be deployed in an operation and maintenance technology network.
In the embodiment of the invention, the data output by the hardware equipment of the operation and maintenance technology network is in seamless connection with the distributed computing system, and the distributed computing system does not need to pay attention to the acquisition of the data output by the hardware equipment of the operation and maintenance technology network, and only needs to issue the computing task according to the service requirement. When a distributed computing system is needed in a certain production environment, the processing device and the distributed computing system for data provided by the embodiment of the present invention may be deployed in the production environment. By combining with a distributed computing system, real-time online diagnosis and fault prediction can be more flexibly and conveniently provided for production line machines, and the industrialization and intelligence levels of factories are improved.
In the embodiment of the invention, the operating environment and software related to the data processing device can be packaged in the Dock image, and when the data processing device provided by the embodiment of the invention is deployed, the operating environment can be installed in a network-free environment only according to the Dock image, so that the trouble of installing the operating environment in the network-free environment is avoided.
The embodiments of the invention have at least the following beneficial effects:
in the embodiment of the invention, each acquisition component can analyze a protocol for transmitting data, and the acquisition component is used for acquiring data output by corresponding hardware equipment, wherein the acquisition component can analyze the protocol for transmitting data of the corresponding hardware equipment, analyze the acquired data, and convert the analyzed data into data in a data format required by the distributed computing system.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other similar elements in a process, method, article, or apparatus that comprises the element.
Finally, it is to be noted that: the above description is only a preferred embodiment of the present invention, and is only used to illustrate the technical solutions of the present invention, and not to limit the protection scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (16)
1. A method for processing data, comprising:
determining a protocol for transmitting data of any hardware device in an operation and maintenance technology network, wherein the operation and maintenance technology network is an industrial communication network for connecting devices and systems in a production field;
determining a target acquisition component capable of analyzing data transmitted by the hardware equipment from at least one preset acquisition component;
calling the target acquisition component to acquire data output by the hardware equipment, and analyzing the data output by the hardware equipment to generate analyzed data;
converting the analyzed data into data in a distributed format, wherein the distributed format is a data format required by a distributed computing system;
and outputting the data in the distributed format to the distributed computing system.
2. The data processing method according to claim 1,
the converting the parsed data into data in a data format required by the distributed computing system includes:
converting the analyzed data into data in a preset standard format;
and converting the data in the standard format into the data in the distributed format.
3. The data processing method according to claim 2,
the processing method comprises the following steps:
caching the data in the standard format into a buffer pool;
and when the data in the standard format is converted into the data in the distributed format, acquiring the data in the standard format from the buffer pool.
4. The data processing method according to claim 3,
the processing method comprises the following steps:
acquiring the data in the standard format from the buffer pool;
converting the data in the standard format into data in a database format;
storing the data in the database format into a database;
when a data request sent by the distributed computing system is received, judging whether data requested by the data request exists in the buffer pool, and if the data requested by the data request does not exist in the buffer pool, acquiring data in a database format requested by the data request from the database;
converting the data in the database format requested by the data request into the data in the distributed format, and outputting the data in the distributed format to the distributed computing system.
5. The data processing method according to claim 1,
each of the acquisition components includes: an interface for a request mode and an interface for a push mode;
before the invoking the target acquisition component to acquire the data output by the hardware device, the processing method further includes:
determining a mode of the hardware device for outputting data;
when the mode of the hardware equipment for outputting data is a request mode, an interface used for requesting the mode in the target acquisition component is realized;
and when the mode of the hardware equipment for outputting data is a push mode, realizing an interface used for the push mode in the target acquisition component.
6. The data processing method according to claim 2,
the converting the data in the standard format into the data in the distributed format includes:
converting the data in the standard format into data in a kafka data format by using a kafka program;
wherein the distributed format comprises: the kafka data format;
the outputting the data in the distributed format to the distributed computing system includes:
inputting the data in the kafka data format into a queue to enable the distributed computing system to obtain the data in the kafka data format from the queue.
7. The data processing method according to claim 1,
the processing method further comprises the following steps:
receiving an externally input command format of an issued message and a command format of a data return value of the acquisition component;
and generating the acquisition component according to the command format of the sent message and the command format of the data return value.
8. The method for processing data according to any one of claims 1 to 7,
the data in the standard format comprises: any one or more of a name of the data, a value of the data, a time at which the data is generated, a type of the value of the data, a unit of the value of the data, and a state of a hardware device that generates the data.
9. A device for processing work data, comprising:
at least one acquisition component (301), an adapter module (302) and a conversion module (303);
each of said acquisition components (301) being capable of resolving a protocol for transmitting data;
the adapter module (302) is configured to determine a protocol for transmitting data of any hardware device in an operation and maintenance technology network, determine a target acquisition component from at least one preset acquisition component (301), and call the target acquisition component, where the operation and maintenance technology network is an industrial communication network for connecting devices and systems in a production field, and the target acquisition component is an acquisition component (301) capable of resolving the protocol for transmitting data of the hardware device;
the acquisition component (301) is configured to acquire data output by the hardware device, analyze the data output by the hardware device, and generate analyzed data;
the conversion module (303) is configured to convert the parsed data into data in a distributed format, and output the data in the distributed format to the distributed computing system, where the distributed format is a data format required by the distributed computing system.
10. The apparatus for processing data according to claim 9,
the conversion module (303) is configured to convert the analyzed data into data in a preset standard format, and convert the data in the standard format into the data in the distributed format.
11. The apparatus for processing data according to claim 10,
the conversion module (303) is further configured to, after converting the parsed data into data in a preset standard format and before converting the parsed data into data in a distributed format, cache the data in the standard format in a buffer pool, and acquire the data in the standard format from the buffer pool.
12. The apparatus for processing data according to claim 11,
further comprising: a data persistence module;
the data persistence module is used for acquiring the data in the standard format from the buffer pool, converting the data in the standard format into the data in the database format, and storing the data in the database format into a database;
the conversion module (303) is further configured to, when a data request sent by the distributed computing system is received, determine whether data requested by the data request exists in the buffer pool, if not, obtain data in a database format requested by the data request from the database, convert the data in the database format requested by the data request into the data in the distributed format, and output the data in the distributed format to the distributed computing system.
13. The apparatus for processing data according to claim 9,
each acquisition assembly (301) comprises: an interface for a request mode and an interface for a push mode;
the adapter module (302) is further configured to determine a mode in which the hardware device outputs data before the target acquisition component is called to acquire the data output by the hardware device, implement an interface for requesting the mode in the target acquisition component when the mode in which the hardware device outputs data is a request mode, and implement an interface for pushing the mode in the target acquisition component when the mode in which the hardware device outputs data is a push mode.
14. The apparatus for processing data according to claim 10,
the conversion module (303) is used for converting the data in the standard format into the data in the kafka data format by using a kafka program, and inputting the data in the kafka data format into a queue so that the distributed computing system acquires the data in the kafka data format from the queue;
wherein the distributed format comprises: the kafka data format.
15. The apparatus for processing data according to claim 9,
further comprising: a component generation module;
the component generating module is used for receiving an externally input command format of an issued message and a command format of a data return value of the acquiring component, and generating the acquiring component (301) according to the command format of the issued message and the command format of the data return value.
16. The apparatus for processing data according to any of claims 9-15, wherein the data in standard format comprises: any one or more of a name of the data, a value of the data, a time at which the data is generated, a type of the value of the data, a unit of the value of the data, and a state of a hardware device that generates the data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910933143.0A CN112579675A (en) | 2019-09-29 | 2019-09-29 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910933143.0A CN112579675A (en) | 2019-09-29 | 2019-09-29 | Data processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112579675A true CN112579675A (en) | 2021-03-30 |
Family
ID=75111367
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910933143.0A Pending CN112579675A (en) | 2019-09-29 | 2019-09-29 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112579675A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113609203A (en) * | 2021-08-20 | 2021-11-05 | 苏州良医汇网络科技有限公司 | Medicine data conversion method and system |
-
2019
- 2019-09-29 CN CN201910933143.0A patent/CN112579675A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113609203A (en) * | 2021-08-20 | 2021-11-05 | 苏州良医汇网络科技有限公司 | Medicine data conversion method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107438095B (en) | Session interface proxy for manufacturing operational information | |
| US11323519B2 (en) | Internet of things pub-sub data publisher | |
| EP3374890B1 (en) | Event stream processing cluster manager | |
| US7142929B2 (en) | Process data management | |
| CN114253569A (en) | Updating operational technical equipment using a container orchestration system | |
| US20190258214A1 (en) | Information processing system and information processing method | |
| CN114253224A (en) | Integrating a container arrangement system with an operating technology device | |
| US20200257698A1 (en) | Data array of objects indexing | |
| US20220083015A1 (en) | Converged machine learning and operational technology data acquisition platform | |
| CN114296405A (en) | Implementation of serverless functionality using container orchestration systems and operating technology devices | |
| CN114490268A (en) | Full link monitoring method, device, equipment, storage medium and program product | |
| CN113190524A (en) | Industrial big data acquisition method and system | |
| CN112579675A (en) | Data processing method and device | |
| CN112817539A (en) | Industrial data storage method and system, electronic device and storage medium | |
| JP7107046B2 (en) | Control system, search device and search program | |
| US11822981B2 (en) | Common gateway platform | |
| KR20170093562A (en) | Smart factory connection module and operating method thereof | |
| US20220101139A1 (en) | System for Action Indication Determination | |
| CN115333943A (en) | Deterministic network resource configuration system, method, device and storage medium | |
| CN114443343A (en) | Method, device, equipment and storage medium for service interface to feed back abnormal information | |
| CN111596628A (en) | Acquisition method and acquisition system for mass industrial data | |
| CN117193225B (en) | Wafer manufacturing process flow control method and system | |
| Nurgaliyev et al. | AN ANALYSIS OF THE HETEROGENEOUS IOT DEVICE NETWORK INTERACTION IN A CYBER-PHYSICAL SYSTEM | |
| CN116860353A (en) | Data processing method, device, electronic equipment and storage medium | |
| CN115567577A (en) | Crane automation data communication system and method and crane |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |