CN114064140B - Fault recording data storage and access method and device and storage medium - Google Patents

Fault recording data storage and access method and device and storage medium Download PDF

Info

Publication number
CN114064140B
CN114064140B CN202111205661.4A CN202111205661A CN114064140B CN 114064140 B CN114064140 B CN 114064140B CN 202111205661 A CN202111205661 A CN 202111205661A CN 114064140 B CN114064140 B CN 114064140B
Authority
CN
China
Prior art keywords
data
compressed
processing
processing plug
plug
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202111205661.4A
Other languages
Chinese (zh)
Other versions
CN114064140A (en
Inventor
王亮
陈龙
侯学勇
龚行梁
李德文
鲁振威
余乾
刘明慧
孔维辉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NR Electric Co Ltd
NR Engineering Co Ltd
Original Assignee
NR Electric Co Ltd
NR Engineering Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by NR Electric Co Ltd, NR Engineering Co Ltd filed Critical NR Electric Co Ltd
Priority to CN202111205661.4A priority Critical patent/CN114064140B/en
Publication of CN114064140A publication Critical patent/CN114064140A/en
Application granted granted Critical
Publication of CN114064140B publication Critical patent/CN114064140B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44Arrangements for executing specific programs
    • G06F9/445Program loading or initiating
    • G06F9/44521Dynamic linking or loading; Link editing at or after load time, e.g. Java class loading
    • G06F9/44526Plug-ins; Add-ons
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the invention discloses a fault recording data storage and access method and device, and a storage medium, wherein the method comprises the following steps: acquiring an original message; determining data to be compressed in a first processing plug-in based on the original message; transmitting the data to be compressed to a second processing plug-in through a first high-speed bus; processing the data to be compressed based on multithreading corresponding to the number of processing cores established through the second processing plug-in, and obtaining compressed data; and storing the compressed data into a second storage space of the second processing plug-in. In the scheme, two processors are respectively used for processing data and compressing the data, so that the data processing speed of the fault wave recording device is improved, and the data processing speed of the fault wave recording device is further improved by utilizing the multithreaded compressed data.

Description

Fault recording data storage and access method and device and storage medium
Technical Field
The invention relates to the technical field of power systems and automation thereof, in particular to a fault recording data storage and access method and device and a storage medium.
Background
The fault wave recording device is a secondary device for relay protection of a transformer substation, and has the main task of recording the change process of relevant system electrical parameters (current, voltage, switching value and the like) before and after the occurrence of large disturbance of a power system and the action behavior of a relay protection and safety automatic device, thereby providing assistance for disturbance analysis and being equivalent to a black box for operation of the transformer substation. Besides extracting current, voltage analog quantity and switching value from the message, the fault wave recording device also needs to record all the original communication messages carrying the adopted numerical values, so that the data processing and storage of the fault wave recording device are problematic.
Disclosure of Invention
The embodiment of the invention provides a fault recording data storage and access method and device and a storage medium, which can solve the problems of compression, storage and access of a large amount of data of an intelligent substation fault recording device.
The technical scheme of the invention is realized as follows:
the embodiment of the invention provides a fault recording data storage and access method, which comprises the following steps:
acquiring an original message;
determining data to be compressed in a first processing plug-in based on the original message;
transmitting the data to be compressed to a second processing plug-in through a first high-speed bus;
processing the data to be compressed based on multithreading corresponding to the number of processing cores established through the second processing plug-in, and obtaining compressed data;
and storing the compressed data into a second storage space of the second processing plug-in.
In the above solution, the processing the data to be compressed based on the multithreading corresponding to the number of processing cores established by the second processing plug-in unit to obtain compressed data includes:
establishing the same number of the multithreading by compressing an application program based on the number of processing cores in the second processing plug-in;
And based on the data to be compressed, performing compression processing through the multithreading to obtain the compressed data.
In the above scheme, the compressing, based on the data to be compressed, by the multithreading to obtain the compressed data includes:
uniformly distributing the data to be compressed to the multithreading to obtain sub-data to be compressed of each thread;
based on the sub-data to be compressed of each thread, respectively performing compression processing to obtain sub-compressed data of each thread;
and obtaining the compressed data by fusing the sub-compressed data of each thread.
In the above solution, the determining, based on the original packet, data to be compressed in the first processing plug-in includes:
preprocessing the original message to determine effective information;
and based on the effective information, adding compression related information to obtain the data to be compressed in the first processing plug-in.
In the above scheme, the preprocessing the original message to determine effective information includes:
classifying according to the type of the original message to obtain a classification result;
and analyzing the data characteristics of the network message based on the classification result, and determining the effective information.
In the above solution, the adding compression related information based on the effective information to obtain the data to be compressed in the first processing plug-in includes:
based on the effective information, adding a compression related information head to obtain complete effective data; wherein the compression related information header includes: data frame length, file name and storage path;
and obtaining the data to be compressed in the first processing plug-in based on the effective data.
In the above scheme, the obtaining the original message includes:
acquiring network message data;
and performing time tag processing on the network message data to obtain the original message.
In the above aspect, after the storing the compressed data in the second storage space of the second processing plug-in, the method further includes:
receiving a second external data access request through the first processing plug-in;
and responding to the second external data access request, and acquiring the compressed data in a second processing plug-in through the first processing plug-in based on network file system NFS service.
In the above solution, the obtaining, based on the network file system NFS service, the compressed data in the second processing plug-in via the first processing plug-in the response to the second external data access request includes:
Responding to the second external data access request, and realizing the NFS service and Remote Procedure Call (RPC) service through a connection channel by the first processing plug-in to access the compressed data in the second processing plug-in; the connection channel is established based on the network protocol of the first processing plug-in and the network protocol of the second processing plug-in.
In the above solution, the responding to the second external data access request, implementing the NFS service and the RPC service through the first processing plug-in through a connection channel, to access the compressed data in the second processing plug-in includes:
responding to the second external data access request, and extracting the compressed data in the second processing plug-in through the NFS service;
and transmitting the compressed data to the first processing plug-in unit through the RPC service and then completing the access of the compressed data in the second processing plug-in unit.
In the above aspect, after the storing the compressed data in the second storage space of the second processing plug-in, the method further includes:
receiving a first external data access request through the first processing plug-in;
And responding to the first external data access request, and accessing the stored data in the first processing plug-in to the local space of the first processing plug-in.
The embodiment of the invention provides a data storage and access device of a fault wave recording device, which is characterized by comprising an acquisition plug-in unit, a first processing plug-in unit and a second processing plug-in unit; wherein,
the acquisition plug-in is used for acquiring an original message;
the first processing plug-in is used for determining data to be compressed in the first processing plug-in based on the original message;
the second processing plug-in is used for processing the data to be compressed based on multithreading which is established by the second processing plug-in and corresponds to the number of processing cores, so as to obtain compressed data; and storing the compressed data into a second storage space of the second processing plug-in.
The embodiment of the invention provides a fault recording data storage and access device, which comprises:
a memory for storing executable instructions;
and a processor for executing the executable instructions stored in the memory, wherein the processor performs the data storage and access method when the executable instructions are executed.
An embodiment of the invention provides a storage medium storing executable instructions that, when executed by one or more processors, perform the data storage and access method.
The embodiment of the invention provides a fault recording data storage and access method and device and a storage medium, wherein the method comprises the following steps: acquiring an original message; determining data to be compressed in a first processing plug-in based on the original message; transmitting the data to be compressed to a second processing plug-in through a first high-speed bus; processing the data to be compressed based on multithreading corresponding to the number of processing cores established through the second processing plug-in, and obtaining compressed data; and storing the compressed data into a second storage space of the second processing plug-in. In the scheme, two processors are respectively used for processing data and compressing the data, so that the data processing speed of the fault wave recording device is improved, and the data processing speed of the fault wave recording device is further improved by utilizing the multithreaded compressed data.
Drawings
FIG. 1 is a schematic diagram of an alternative fault recording device for a fault recording data storage and access method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of an alternative internal structure of a method for storing and accessing fault recording data according to an embodiment of the present invention;
FIG. 3 is a schematic flow chart of an alternative method for storing and accessing fault recording data according to an embodiment of the present invention;
FIG. 4 is a flowchart of an optional internal processing compressed data of a fault record data storage and access method according to an embodiment of the present invention;
FIG. 5 is a schematic flow chart of an alternative method for storing and accessing fault recording data according to an embodiment of the present invention;
FIG. 6 is a schematic flow chart of an alternative method for storing and accessing fault recording data according to an embodiment of the present invention;
FIG. 7 is a schematic flow chart of an alternative method for storing and accessing fault recording data according to an embodiment of the present invention;
FIG. 8 is a schematic flow chart of an alternative method for storing and accessing fault record data according to an embodiment of the present invention;
FIG. 9 is a schematic diagram of an alternative external access data flow of a method for storing and accessing fault record data according to an embodiment of the present invention;
FIG. 10 is a schematic flow chart of an alternative method for storing and accessing fault record data according to an embodiment of the present invention;
fig. 11 is a schematic structural diagram of a fault recording data storage and access device according to an embodiment of the present invention;
Fig. 12 is a schematic structural diagram of a fault recording data storage and access device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without making any inventive effort are within the scope of the present invention.
In order to better understand the aspects of the present invention, the present invention will be described in further detail with reference to the accompanying drawings and detailed description. The schematic diagram of the fault recording device with the optional fault recording data storage and access method provided by the embodiment of the invention is shown in fig. 1, wherein the schematic diagram comprises an acquisition plug-in unit, a first processing plug-in unit, a first high-speed bus, a second processing plug-in unit and external data access. The acquisition plug-in mainly comprises a Field Programmable Gate Array (FPGA), and is mainly used for carrying out packet capturing and time mark marking processing on message data to obtain original data (equivalent to an original message); the second high-speed bus mainly transmits the original data to the first processing plug-in; the first processing plug-in unit is used for preprocessing the original data mainly through an original data processing process to obtain data to be compressed, and sending the data through a data sending process to be compressed; the first high-speed bus mainly transmits data to be compressed to the second processing plug-in unit; the second processing plug-in mainly carries out compression processing on data to be compressed through a compression storage control process to obtain compressed data. 1-5 in FIG. 1, and external data access is mainly 6-7 in FIG. 1, wherein the first processing plug-in acquires compressed data of a second storage space in the second processing plug-in through a network file system; and secondly, the first processing plug-in acquires the storage data of the local first storage space. The internal structure results of the first processing plug-in and the second processing plug-in are the same, the internal structures of the first processing plug-in and the second processing plug-in are shown in figure 2, and the first processing plug-in and the second processing plug-in comprise a CPU, an FPGA, a storage medium, an Ethernet interface and a backboard plug-in, wherein the FPGA and the backboard plug-in are connected through an Ethernet or a high-speed bus; the Ethernet interfaces have 8 paths in total, and the transmission speed is hundred megabytes or kilomega.
In some embodiments of the present invention, fig. 3 is a schematic flow chart illustrating an alternative method for storing and accessing fault recording data according to an embodiment of the present invention, and will be described with reference to the steps shown in fig. 3.
S101, acquiring an original message.
In some embodiments of the present invention, the fault recording data storage and access method is suitable for a scenario of processing a large amount of data of an intelligent substation, the fault recording device is equivalent to a terminal, and embodiments of the present invention are not limited.
The fault wave recording device is a secondary device for relay protection of a transformer substation, and has the main task of recording the change process of relevant system electrical parameters (current, voltage, switching value and the like) before and after the occurrence of large disturbance of a power system and the action behavior of a relay protection and safety automatic device, thereby providing assistance for disturbance analysis and being equivalent to a black box for operation of the transformer substation.
In some embodiments of the present invention, the terminal may collect network message data through a collecting plug-in, and obtain an original message after preprocessing, where the collecting plug-in is composed of an FPGA and a transmission bus.
A message (message) is a data unit exchanged and transmitted in a network, i.e. a data block to be sent by a station at one time. The message contains the complete data information to be sent, and the length of the message is not consistent, and the length of the message is unlimited and variable.
The original message in the embodiment of the invention can be a sampling value SV message, general object-oriented substation event GOOSE message data and the like.
In some embodiments of the present invention, step S101 may be implemented by S1011-S1012 as follows:
s1011, acquiring network message data.
In some embodiments of the invention, the network packet data includes SV packets, GOOSE packets, and other network packet data.
The sample value SV message is mainly used for exchanging related model objects and services of sample values in a sample data set and mapping between the model objects and services to ISO/IEC8802-3 frames. The GOOSE message is mainly used for communication between process layer equipment and spacer layer equipment, and the content of the GOOSE message comprises primary equipment body position/alarm information, merging unit/intelligent terminal self-checking information, protection tripping/reclosing information and measurement and control remote control switching-on/switching-off/locking information which are transmitted between a merging unit, an intelligent terminal and devices such as protection, measurement and control, fault recording and the like.
In some embodiments of the present invention, the terminal may obtain SV messages, GOOSE messages, and other network message data through the acquisition plug-in.
S1012, performing time-stamp processing on the network message data to obtain an original message.
In some embodiments of the present invention, the terminal may capture packets of the acquired network packet data through the acquisition plug-in, and time tags the network packet data left after capturing the packets.
It will be appreciated that in some embodiments of the present invention, the terminal may obtain the original message through the acquisition plug-in, providing a basis for subsequent data storage and access.
S102, determining data to be compressed in a first processing plug-in based on the original message.
In some embodiments of the present invention, the first processing plug-in is composed of main components such as a multi-core CPU, a storage medium, and a high-speed bus, and functions to receive the preprocessed original message pushed by the collecting plug-in. The data to be compressed is to sort and arrange the original message to be compressed and stored, extract the effective information, add the compression related information and organize the whole data frame.
In some embodiments of the present invention, the terminal may transmit the original packet in the acquisition plug-in to the first processing plug-in; the first processing plug-in carries out effective information extraction processing on the original message, and data to be compressed in the first processing plug-in are obtained.
S103, transmitting the data to be compressed to the second processing plug-in unit through the first high-speed bus.
In some embodiments of the present invention, the second processing plug-in is composed of main components such as a multi-core CPU, a storage medium, a high-speed bus, and the like, and functions to receive data to be compressed and stored pushed by the CPU1 plug-in.
In some embodiments of the present invention, the terminal may transmit data to be compressed in the first processing plug-in to the second processing plug-in through the first high-speed bus; the second processing plug-in obtains data to be compressed in the first processing plug-in.
Illustratively, fig. 4 is a flowchart of an optional internal processing compressed data flow of a fault record data storage and access method, and as shown in fig. 4, the first high-speed bus transfers data to be compressed in the first processing plug-in to the second processing plug-in.
And S104, processing the data to be compressed based on the multithreading corresponding to the number of the processing cores established through the second processing plug-in, and obtaining compressed data.
In some embodiments of the present invention, multithreading adopts a multithreading interface of a Linux operating system, and compressed data is data of compressed data to be compressed, so that storage is convenient, and storage space is saved.
In some embodiments of the present invention, the terminal may establish multiple threads through the number of processing cores in the second processing plug-in, where the number of threads of multiple threads is consistent with the number of processing cores, so as to ensure that multiple threads can process in parallel on the processing cores bound respectively, and process data to be compressed quickly, thereby obtaining compressed data.
Illustratively, the number of processing cores in the second processing plug-in is 8, as is the number of multithreading established.
S105, storing the compressed data into a second storage space of a second processing plug-in.
In some embodiments of the invention, the second memory space is located within the second processing card, specifically for storing data.
In some embodiments of the invention, the terminal may store the compressed data to a second memory space of a second processing plug-in.
It may be appreciated that, in some embodiments of the present invention, the terminal may obtain the original message; determining data to be compressed in a first processing plug-in based on the original message; transmitting the data to be compressed to a second processing plug-in through a first high-speed bus; processing the data to be compressed based on multithreading corresponding to the number of processing cores established through the second processing plug-in, and obtaining compressed data; and storing the compressed data into a second storage space of the second processing plug-in unit, so that the data processing speed of the fault wave recording device is improved.
In some embodiments of the present invention, referring to fig. 5, fig. 5 is a schematic flow chart of an alternative method for storing and accessing fault recording data according to an embodiment of the present invention, based on multithreading corresponding to the number of processing cores established by the second processing plug-in, processing data to be compressed, and obtaining compressed data may be implemented by S201-S202, which will be described with reference to the steps shown in fig. 5.
S201, based on the number of processing cores in the second processing plug-in, the same number of multithreading is established through the compression application program.
In some embodiments of the present invention, the terminal may notify the upper application program through the received data to be compressed by the lower layer receiving driver, and start to create threads through the compression application program, where the number of threads is consistent with the number of processing cores.
S202, compressing the data to be compressed through multithreading to obtain compressed data.
In some embodiments of the present invention, compression employs an LZ77 (Lempel-Ziv-1977) based compression algorithm with a Huffman compression algorithm.
Both huffman coding and LZ77 (Lempel-Ziv-1977) are lossless compression, where the huffman compression algorithm is compression with minimal redundancy coding, and the LZ77 compression algorithm is compression with dictionary.
In some embodiments of the present invention, the terminal may perform compression processing on data to be compressed by using a compression algorithm through multithreading to obtain compressed data.
Illustratively, as shown in the second processing plug-in part of fig. 4, the second processing plug-in receives data to be compressed of the first plug-in, starts a compression application program according to the data to be compressed, establishes multithreading, binds the multithreading with the processing core, and performs compression processing through n compression storage threads (multithreading) to obtain compressed data.
It can be appreciated that in some embodiments of the present invention, the terminal may perform compression processing on data to be compressed by establishing multiple threads to obtain compressed data, and the processing speed of the data is improved by using the multiple threads to perform parallel processing, so as to provide conditions for subsequent data storage and access.
In some embodiments of the present invention, referring to fig. 6, fig. 6 is a schematic flow chart of an alternative method for storing and accessing fault recording data according to an embodiment of the present invention, and step S202 may be implemented by steps S2021 to S2023, which will be described with reference to the steps shown in fig. 6.
S2021, uniformly distributing the data to be compressed to multiple threads to obtain sub-data to be compressed of each thread.
In some embodiments of the present invention, the terminal may uniformly distribute the data to be compressed to each of the multiple threads according to the number of the multiple threads, so that each thread obtains the sub-data to be compressed to be processed.
For example, the data to be compressed contains 240 pieces of valid information, and the number of the multithreading is 8, so that the sub-data to be compressed processed by each thread contains 30 pieces of valid information respectively.
S2022, based on the sub-compressed data of each thread, respectively performing compression processing to obtain sub-compressed data of each thread.
In some embodiments of the present invention, the terminal may perform compression processing on the sub-compressed data of each thread through a compression algorithm, to obtain sub-compressed data of each thread.
S2023, sub-compressed data of each thread are subjected to data fusion to obtain compressed data.
In some embodiments of the present invention, the terminal may obtain compressed data of the data to be compressed by performing data fusion on the sub-compressed data of each thread.
For example, the data to be compressed includes 240 pieces of effective information, the number of the multiple threads is 8, the sub-data to be compressed processed by each thread includes 30 pieces of effective information, 1 piece of sub-compressed data is obtained after each thread performs compression processing, and the 8 pieces of sub-compressed data are subjected to data fusion to obtain 1 piece of compressed data, namely compressed data of the data to be compressed.
It can be appreciated that in some embodiments of the present invention, the terminal may obtain sub-data to be compressed for each thread by uniformly distributing the data to be compressed to multiple threads; based on the sub-compressed data of each thread, respectively performing compression processing to obtain sub-compressed data of each thread; the sub-compressed data of each thread is subjected to data fusion to obtain compressed data, so that the data processing speed is improved.
In some embodiments of the present invention, referring to fig. 7, fig. 7 is a schematic flow chart of an alternative method for storing and accessing fault recording data provided in the embodiment of the present invention, and based on an original packet, determining that data to be compressed in a first processing plug-in may be implemented by S301-S302, which will be described with reference to the steps shown in fig. 7.
S301, preprocessing an original message and determining effective information.
In some embodiments of the present invention, the terminal may perform a classification preprocessing on the original packet through the first processing plug-in, and select valid information from the original packet.
In some embodiments of the present invention, step S301 may be implemented by S3011 and S3012 as follows:
s3011, classifying according to the type of the original message to obtain a classification result.
In some embodiments of the present invention, the types of the original message include: SV messages, GOOSE messages, and other network message data.
In some embodiments of the present invention, the terminal may classify the original message by using different message types to which the original message belongs, and separate the messages of the same type in the original message together, thereby separating the original message into several types of messages.
S3012, analyzing the data characteristics of the network message based on the classification result, and determining effective information.
In some embodiments of the present invention, the network message data features include: network message flow, network message quality, analog quantity and switching value in network message.
In some embodiments of the present invention, the terminal may perform data feature analysis on each type of message data through the result of the original message classification, extract valid data, and remove header and trailer information of the data, thereby determining valid information.
It can be understood that, in some embodiments of the present invention, the terminal may classify according to the type of the original message, to obtain a classification result; based on the classification result, analyzing the data characteristics of the network message, determining effective information, and removing ineffective information until effective information is stored, thereby improving the effectiveness of data storage and saving the storage space.
S302, based on the effective information, adding compression related information to obtain data to be compressed in the first processing plug-in.
In some embodiments of the invention, the compression related information comprises: data frame length, file name, storage path.
In some embodiments of the present invention, the terminal may obtain the data to be compressed in the first processing plug-in by adding compression related information to the effective information.
Illustratively, fig. 4 is a schematic diagram of an alternative data storage method for storing and accessing fault recording data. As can be seen, the data store comprises a data processing portion of the first processing plug-in and a data compression portion of the second processing plug-in. The first processing plug-in classifies the received message data, removes message header and message tail information, extracts effective data 1, effective data 2 and … and effective data N (corresponding to effective information), and obtains data to be compressed by adding a message header (corresponding to compression related information header) to the effective data 1-N.
It may be appreciated that in some embodiments of the present invention, the terminal may determine the valid information by preprocessing the original message; based on the effective information, the compression related information is added to obtain the data to be compressed in the first processing plug-in, and the effective information of the original message is extracted, so that the effectiveness of data processing is improved, and the storage space occupied by data storage is reduced.
In some embodiments of the present invention, step S302 may be implemented by S3021 and S3022 as follows:
s3021, based on the effective information, adding a compression related information head to obtain complete effective data; wherein the compression related information header includes: data frame length, file name, storage path.
In some embodiments of the present invention, the terminal may obtain the complete valid data by adding a compression related information header based on the valid information, where the compression related information header includes a data frame length, a file name, and a storage path.
And S3022, obtaining data to be compressed in the first processing plug-in based on the effective data.
In some embodiments of the present invention, the terminal may obtain the data to be compressed in the first processing plug-in through the valid data. The added effective information of the compression related information head is effective data, namely the data to be compressed in the first processing plug-in.
It may be appreciated that in some embodiments of the present invention, the terminal may obtain complete valid data by adding a compression related information header based on the valid information; and obtaining the data to be compressed in the first processing plug-in based on the effective data, and providing the data to be compressed for the compression processing of the subsequent second processing plug-in.
In some embodiments of the present invention, referring to fig. 8, fig. 8 is a schematic flow chart of an alternative method for storing and accessing fault recording data according to an embodiment of the present invention, and after step S105, S106 to S107 are further included. The steps shown in fig. 8 will be described.
S106, receiving a second external data access request through the first processing plug-in.
In some embodiments of the invention, the second external data access request refers to a request to access compressed data within the second processing plug-in.
In some embodiments of the present invention, the terminal may receive an external data access request through the first processing plug-in, i.e. a request to access compressed data inside the second processing plug-in.
S107, compressed data in the second processing plug-in is acquired through the first processing plug-in based on the network file system NFS service in response to the second external data access request.
In some embodiments of the present invention, the network file system, english Network File System (NFS), is a UNIX presentation layer protocol (presentation layer protocol) developed by SUN corporation, which enables users to access files elsewhere on the network as if they were using their own computer. NFS is one of the currently mainstream heterogeneous platform shared file systems. The method is mainly applied to the UNIX/LINUX environment. Originally developed by Sun Microsystems, the system can support file sharing among different types of systems through a network, is widely applied to heterogeneous operating system platforms such as FreeBSD, SCO, solaris, and allows one system to share directories and files with other people on the network. By using NFS, users and programs can access files on a remote system like local files, enabling nodes of each computer to use resources on the network as conveniently as local resources. NFS can be used for remote access and sharing of network files in different types of computers, operating systems, network architectures, and transport protocol operating environments.
In some embodiments of the present invention, NFS operates on the principle of using a client/server architecture, consisting of one client program and one server program. The server program provides access to the file system to other computers, a process called export. NFS client programs "transport" shared file systems out of NFS servers when they access them. NFS transport protocol is used for file access and shared communication between a server and a client, thereby enabling the client to remotely access data stored on a storage device. The first processing plug-in is equivalent to a client, and the second processing plug-in is equivalent to a server.
In some embodiments of the present invention, the terminal may obtain, by the first processing plug-in, compressed data within the second processing plug-in by deploying network file systems NFS on the first processing plug-in and the second processing plug-in, respectively, and then by the NFS service.
It will be appreciated that in some embodiments of the invention, the terminal may receive a second external data access request through the first processing plug-in; and responding to the second external data access request, acquiring the compressed data in the second processing plug-in through the first processing plug-in based on the network file system NFS service, and realizing external access to the compressed data.
In some embodiments of the present invention, in response to the second external data access request, obtaining, via the first processing plug-in, the compressed data within the second processing plug-in based on the network file system NFS service may be implemented by S1071 as follows:
s1071, responding to a second external data access request, and realizing NFS service and Remote Procedure Call (RPC) service through a first processing plug-in unit by a connection channel to access compressed data in the second processing plug-in unit; the connection channel is established based on the network protocol of the first processing plug-in and the network protocol of the second processing plug-in.
In some embodiments of the present invention, NFS is a UDP/IP protocol based application implemented primarily using remote procedure call RPC mechanisms, which provide a set of operations to access remote files independent of machine, operating system, and low-level transport protocols. The RPC employs XDR support. XDR is a machine-independent protocol for data description encoding that encodes and decodes data transmitted over the network in a format that is independent of any machine architecture, supporting the transfer of data between heterogeneous systems.
In some embodiments of the invention, the network protocol uses the TCP/IP protocol. The TCP/IP protocol, the Transmission control/network protocol, is also known as the network communication protocol. It is the most basic communication protocol in use of the network. The TCP/IP transport protocol specifies the standards and methods for communicating portions of the internet. And, TCP/IP transmission protocol is two important protocols that guarantee the timely, complete transmission of network data information. The TCP/IP transport protocol is strictly a four-layer architecture comprising: an application layer, a transport layer, a network layer, and a data link layer. The main protocols of the application layer include Telnet, FTP, SMTP, etc. for receiving data from the transmission layer or transmitting data to the transmission layer according to different application requirements and modes; the main protocols of the transmission layer include UDP and TCP, which are channels for combining the user platform with the internal data of the computer information network, so that the data transmission and the data sharing can be realized; the main protocol of the network layer is ICMP, IP, IGMP, which is mainly responsible for the transmission of data packets in the network; the network access layer, also called network interface layer or data link layer, has ARP and RARP as main protocol, and has the main functions of providing link management error detection, effectively processing the information detail problems related to different communication media, etc.
In some embodiments of the present invention, the terminal may access the compressed data in the second processing plug-in through the RPC service in the first processing plug-in, through the connection channel, and through the NFS service in the second processing plug-in.
In some embodiments of the present invention, step S1071 may be implemented by S10711 and S10712 as follows:
s10711, in response to the second external data access request, extracting, by the NFS service, the compressed data in the second processing plug-in.
In some embodiments of the present invention, the terminal may perform local file access through the NFS service, thereby extracting the compressed data in the second processing plug-in.
And S10712, transmitting the compressed data to the first processing plug-in unit through the RPC service, and thus completing the access of the compressed data in the second processing plug-in unit.
In some embodiments of the present invention, the terminal may transfer the extracted compressed data to the RPC service in the first processing plug-in through the connection channel by using the RPC service in the second processing plug-in, so as to complete access of the compressed data in the second processing plug-in.
Fig. 9 is a schematic diagram of an optional external access data flow of a fault recording data storage and access method according to an embodiment of the present invention. As shown in fig. 9, the terminal receives a second external data access request through a first processing plug-in, and the RPC service in the first processing plug-in transmits the access request to the second processing plug-in through a TCP/IP protocol connection channel; the NFS service of the second processing plug-in obtains the compressed data in the second storage space through local file access; the second processing plug-in transmits the compressed data to the first processing plug-in through the NFS service, the RPC service of the second processing plug-in and the TCP/IP protocol connection channel; the external device obtains the compressed data through the RPC service of the first processing plug-in.
It will be appreciated that in some embodiments of the invention, the terminal may extract compressed data in the second processing plug-in by the NFS service in response to the second external data access request; and transmitting the compressed data to the first processing plug-in unit through the RPC service, thereby completing the access of the compressed data in the second processing plug-in unit and completing the external access of the compressed data in the second processing plug-in unit.
In some embodiments of the present invention, referring to fig. 10, fig. 10 is a schematic flow chart of an alternative method for storing and accessing fault recording data according to an embodiment of the present invention, and after step S105, S108 to S109 are further included. The steps shown in fig. 10 will be described.
S108, receiving a first external data access request through the first processing plug-in.
In some embodiments of the present invention, the first external data access request refers to a request to access stored data internal to the first processing plug-in.
In some embodiments of the invention, the terminal may receive, through the first processing plug-in, a request to access stored data inside the first processing plug-in.
S109, responding to a first external data access request, and accessing the storage data in the first processing plug-in by going to the local space of the first processing plug-in.
In some embodiments of the invention, the stored data within the first processing plug-in is data generated by the first processing plug-in itself.
In some embodiments of the present invention, the terminal may perform local file access through the first processing plug-in, thereby accessing the stored data in the first processing plug-in.
Illustratively, FIG. 9 is a schematic diagram of an alternative external access data flow for a fault record data storage and access method. As shown in fig. 9, the terminal receives a first external data access request through the first processing plug-in, and the external device may obtain the storage data of the first storage space by the first processing plug-in directly accessing the local file.
It will be appreciated that in some embodiments of the invention, the terminal may receive a first external data access request through the first processing plug-in; and responding to the first external data access request, accessing the storage data in the first processing plug-in by going to the local space of the first processing plug-in, and completing external access to the storage data of the first processing plug-in.
Based on the foregoing method for storing and accessing fault recording data in the embodiment, the embodiment of the present invention further provides a device for storing and accessing fault recording data, as shown in fig. 11, fig. 11 is a schematic diagram of a structure for storing and accessing fault recording data, where the device includes: an acquisition plug-in 1101, a first processing plug-in 1102, and a second processing plug-in 1103; wherein,
The collecting plug-in 1101 is configured to obtain an original message;
the first processing plug-in 1102 is configured to determine data to be compressed in the first processing plug-in based on the original packet;
the first processing plug-in 1102 is configured to transmit the data to be compressed to a second processing plug-in through a first high-speed bus;
the second processing plug-in 1103 is configured to process the data to be compressed based on the multithread corresponding to the number of processing cores and established by the second processing plug-in, so as to obtain compressed data;
the second processing plug-in 1103 is configured to store the compressed data into a second memory space of the second processing plug-in.
In some embodiments of the present invention, the second processing plug-in 1103 is configured to establish the same number of the multithreading by compressing an application program based on the number of processing cores in the second processing plug-in; and based on the data to be compressed, performing compression processing through the multithreading to obtain the compressed data.
In some embodiments of the present invention, the second processing plug-in 1103 is configured to uniformly distribute the data to be compressed to the multiple threads, so as to obtain sub-data to be compressed of each thread; based on the sub-data to be compressed of each thread, respectively performing compression processing to obtain sub-compressed data of each thread; and obtaining the compressed data by fusing the sub-compressed data of each thread.
In some embodiments of the present invention, the first processing plug-in 1102 is configured to perform preprocessing on the original packet to determine valid information; and based on the effective information, adding compression related information to obtain the data to be compressed in the first processing plug-in.
In some embodiments of the present invention, the first processing plug-in 1102 is configured to classify according to the type of the original packet, so as to obtain a classification result; and analyzing the data characteristics of the network message based on the classification result, and determining the effective information.
In some embodiments of the present invention, the first processing plug-in 1102 is configured to add a compression related information header based on the validity information to obtain complete validity data; wherein the compression related information header includes: data frame length, file name and storage path; and obtaining the data to be compressed in the first processing plug-in based on the effective data.
In some embodiments of the present invention, the collecting plug-in 1101 is configured to obtain network message data; and performing time tag processing on the network message data to obtain the original message.
In some embodiments of the present invention, the first processing plug-in 1102 is configured to receive a second external data access request through the first processing plug-in; and responding to the second external data access request, and acquiring the compressed data in a second processing plug-in through the first processing plug-in based on network file system NFS service.
In some embodiments of the present invention, the first processing plug-in 1102 is configured to implement, in response to the second external data access request, the NFS service and a remote procedure call RPC service through a connection channel via the first processing plug-in to access the compressed data in the second processing plug-in; the connection channel is established based on the network protocol of the first processing plug-in and the network protocol of the second processing plug-in.
In some embodiments of the present invention, the second processing plug-in 1103 is configured to respond to the second external data access request by serving, through the NFS, the compressed data extracted in the second processing plug-in; and transmitting the compressed data to the first processing plug-in unit through the RPC service and then completing the access of the compressed data in the second processing plug-in unit.
In some embodiments of the present invention, the first processing plug-in 1102 is configured to receive a first external data access request through the first processing plug-in; and responding to the first external data access request, and accessing the stored data in the first processing plug-in to the local space of the first processing plug-in.
Based on the foregoing method for storing and accessing fault recording data in the embodiment, the embodiment of the present invention further provides a device for storing and accessing fault recording data, as shown in fig. 12, fig. 12 is a schematic structural diagram two of a device for storing and accessing fault recording data, provided in the embodiment of the present invention, where the device includes: a processor 1201 and a memory 1202; the memory 1201 stores one or more programs executable by the processor, which when executed, perform any of the data storage and access methods of the embodiments described above by the processor 1202.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention.

Claims (11)

1. A fault recording data storage and access method is characterized by comprising the following steps of
Acquiring an original message through an acquisition plug-in;
determining data to be compressed in a first processing plug-in based on the original message;
Transmitting the data to be compressed to a second processing plug-in through a first high-speed bus;
processing the data to be compressed based on multithreading corresponding to the number of processing cores established through the second processing plug-in, and obtaining compressed data;
storing the compressed data to a second storage space of the second processing plug-in;
the compressing processing is performed through the multithreading based on the data to be compressed to obtain the compressed data, including:
uniformly distributing the data to be compressed to the multithreading to obtain sub-data to be compressed of each thread;
based on the sub-data to be compressed of each thread, respectively performing compression processing to obtain sub-compressed data of each thread;
the sub-compressed data of each thread is subjected to data fusion to obtain the compressed data;
after the storing the compressed data in the second storage space of the second processing plug-in, the method further includes:
receiving a second external data access request through the first processing plug-in;
acquiring the compressed data in a second processing plug-in via the first processing plug-in based on a Network File System (NFS) service in response to the second external data access request;
Said obtaining, via said first processing plug-in, said compressed data within a second processing plug-in based on a network file system NFS service in response to said second external data access request, comprising:
extracting, by the NFS service, the compressed data in the second processing plug-in response to the second external data access request;
and transmitting the compressed data to the first processing plug-in unit through the RPC service and then completing the access of the compressed data in the second processing plug-in unit.
2. The method according to claim 1, wherein said processing the data to be compressed based on the multithreading corresponding to the number of processing cores established by the second processing plug-in to obtain compressed data comprises:
establishing the same number of the multithreading by compressing an application program based on the number of processing cores in the second processing plug-in;
and based on the data to be compressed, performing compression processing through the multithreading to obtain the compressed data.
3. The method according to claim 1 or 2, wherein determining data to be compressed in the first processing plug-in based on the original message comprises:
Preprocessing the original message to determine effective information;
and based on the effective information, adding compression related information to obtain the data to be compressed in the first processing plug-in.
4. A method according to claim 3, wherein the preprocessing the original message to determine valid information includes:
classifying according to the type of the original message to obtain a classification result;
and analyzing the data characteristics of the network message based on the classification result, and determining the effective information.
5. A method according to claim 3, wherein said adding compression related information based on said effective information to obtain said data to be compressed in the first processing plug-in comprises:
based on the effective information, adding a compression related information head to obtain complete effective data; wherein the compression related information header includes: data frame length, file name and storage path;
and obtaining the data to be compressed in the first processing plug-in based on the effective data.
6. The method according to claim 1 or 2, wherein the obtaining the original message includes:
acquiring network message data;
And performing time tag processing on the network message data to obtain the original message.
7. The method of claim 1, wherein said obtaining said compressed data within a second processing plug-in via said first processing plug-in based on a network file system NFS service in response to said second external data access request comprises:
responding to the second external data access request, and realizing the NFS service and Remote Procedure Call (RPC) service through a connection channel by the first processing plug-in to access the compressed data in the second processing plug-in; the connection channel is established based on the network protocol of the first processing plug-in and the network protocol of the second processing plug-in.
8. The method according to claim 1 or 2, wherein after said storing said compressed data in a second memory space of said second processing plug-in, said method further comprises:
receiving a first external data access request through the first processing plug-in;
and responding to the first external data access request, and accessing the stored data in the first processing plug-in to the local space of the first processing plug-in.
9. The fault recording data storage and access device is characterized by comprising an acquisition plug-in, a first processing plug-in and a second processing plug-in; wherein,
the acquisition plug-in is used for acquiring an original message through the acquisition plug-in;
the first processing plug-in is used for determining data to be compressed in the first processing plug-in based on the original message;
the second processing plug-in is used for processing the data to be compressed based on multithreading which is established by the second processing plug-in and corresponds to the number of processing cores, so as to obtain compressed data; storing the compressed data to a second storage space of the second processing plug-in;
the second processing plug-in is further configured to uniformly distribute the data to be compressed to the multiple threads, so as to obtain sub-data to be compressed of each thread; based on the sub-data to be compressed of each thread, respectively performing compression processing to obtain sub-compressed data of each thread; the sub-compressed data of each thread is subjected to data fusion to obtain the compressed data;
the first processing plug-in is further configured to receive a second external data access request through the first processing plug-in; acquiring the compressed data in a second processing plug-in via the first processing plug-in based on a Network File System (NFS) service in response to the second external data access request;
The second processing plug-in is further configured to extract, by the NFS service, the compressed data in the second processing plug-in response to the second external data access request; and transmitting the compressed data to the first processing plug-in unit through the RPC service and then completing the access of the compressed data in the second processing plug-in unit.
10. A fault log data storage and access device, comprising:
a memory for storing executable instructions;
a processor for implementing the method of any one of claims 1 to 8 when executing executable instructions stored in said memory.
11. A storage medium having stored thereon executable instructions for causing a processor to perform the method of any one of claims 1 to 8.
CN202111205661.4A 2021-10-15 2021-10-15 Fault recording data storage and access method and device and storage medium Active CN114064140B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111205661.4A CN114064140B (en) 2021-10-15 2021-10-15 Fault recording data storage and access method and device and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111205661.4A CN114064140B (en) 2021-10-15 2021-10-15 Fault recording data storage and access method and device and storage medium

Publications (2)

Publication Number Publication Date
CN114064140A CN114064140A (en) 2022-02-18
CN114064140B true CN114064140B (en) 2024-03-15

Family

ID=80234702

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111205661.4A Active CN114064140B (en) 2021-10-15 2021-10-15 Fault recording data storage and access method and device and storage medium

Country Status (1)

Country Link
CN (1) CN114064140B (en)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102118293A (en) * 2009-12-30 2011-07-06 上海可鲁***软件有限公司 Method for compressing and storing communication messages
CN105007329A (en) * 2015-07-31 2015-10-28 中国航空无线电电子研究所 PCIe-based network sharing storage system and storage method thereof
CN108141471A (en) * 2016-04-07 2018-06-08 华为技术有限公司 The method, apparatus and equipment of compressed data
WO2019119336A1 (en) * 2017-12-21 2019-06-27 深圳大学 Multi-thread compression and decompression methods in generic data gz format, and device
CN110196836A (en) * 2019-03-29 2019-09-03 腾讯科技(深圳)有限公司 A kind of date storage method and device
WO2020232682A1 (en) * 2019-05-22 2020-11-26 深圳大学 Parallel compression system and method employing hardware
CN113138969A (en) * 2021-04-30 2021-07-20 青岛盛世影云影视科技有限公司 Data transmission method and device, electronic equipment and computer readable storage medium

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102118293A (en) * 2009-12-30 2011-07-06 上海可鲁***软件有限公司 Method for compressing and storing communication messages
CN105007329A (en) * 2015-07-31 2015-10-28 中国航空无线电电子研究所 PCIe-based network sharing storage system and storage method thereof
CN108141471A (en) * 2016-04-07 2018-06-08 华为技术有限公司 The method, apparatus and equipment of compressed data
WO2019119336A1 (en) * 2017-12-21 2019-06-27 深圳大学 Multi-thread compression and decompression methods in generic data gz format, and device
CN110196836A (en) * 2019-03-29 2019-09-03 腾讯科技(深圳)有限公司 A kind of date storage method and device
WO2020232682A1 (en) * 2019-05-22 2020-11-26 深圳大学 Parallel compression system and method employing hardware
CN113138969A (en) * 2021-04-30 2021-07-20 青岛盛世影云影视科技有限公司 Data transmission method and device, electronic equipment and computer readable storage medium

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
智能变电站网络分析与故障录波一体化***设计与实现;何君 等;电气技术(08);第160-163页 *
面向IEEE COMTRADE格式的海量录波数据并行压缩/解压算法;桂勋;;电力自动化设备(05);第141-148页 *

Also Published As

Publication number Publication date
CN114064140A (en) 2022-02-18

Similar Documents

Publication Publication Date Title
CN107911237B (en) DPDK-based rapid detection method for data packets in user space
CN101997925A (en) Server monitoring method with early warning function and system thereof
CN112350854B (en) Flow fault positioning method, device, equipment and storage medium
CN113225339B (en) Network security monitoring method and device, computer equipment and storage medium
CN103324128B (en) Primary equipment fault warning comprehensive compression method in power dispatching automation system
CN111222547B (en) Traffic feature extraction method and system for mobile application
CN110928934A (en) Data processing method and device for business analysis
CN113613287A (en) Automatic data acquisition system based on edge calculation
CN114465741B (en) Abnormality detection method, abnormality detection device, computer equipment and storage medium
CN112565231B (en) Safety II area and III area data synchronization method and device based on power dispatching data network
CN110837455B (en) Script-based application configuration information acquisition method and device
RU2668394C2 (en) Packet collection method and system, network device and network management centre
CN114064140B (en) Fault recording data storage and access method and device and storage medium
CN117130870B (en) Transparent request tracking and sampling method and device for Java architecture micro-service system
CN117453280B (en) Code topology and service topology generation method, device, equipment and medium
CN108287874B (en) DB2 database management method and device
CN109040086A (en) A kind of industrial control system DDOS attack emulation mode and device
CN111245880B (en) Behavior trajectory reconstruction-based user experience monitoring method and device
CN109995589B (en) Log collection method and system
CN110166561B (en) Data processing method, device, system, equipment and medium for wearable equipment
CN115766278B (en) Firewall policy generation method, device, equipment and storage medium
CN116074388B (en) Flow forwarding method and system based on log queue
CN112929362B (en) Probe device, front-end message processing method and wireless communication management system
CN116302711B (en) Disaster recovery method and device based on cloud platform mirror image, electronic equipment and storage medium
CN113920698B (en) Early warning method, device, equipment and medium for interface abnormal call

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant