CN110532612A - The operation data processing method and processing device of ship power system - Google Patents
The operation data processing method and processing device of ship power system Download PDFInfo
- Publication number
- CN110532612A CN110532612A CN201910681803.0A CN201910681803A CN110532612A CN 110532612 A CN110532612 A CN 110532612A CN 201910681803 A CN201910681803 A CN 201910681803A CN 110532612 A CN110532612 A CN 110532612A
- Authority
- CN
- China
- Prior art keywords
- operation data
- power system
- ship
- file
- parsing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 24
- 238000012545 processing Methods 0.000 title claims abstract description 18
- 238000004458 analytical method Methods 0.000 claims abstract description 29
- 238000012360 testing method Methods 0.000 claims abstract description 28
- 230000008676 import Effects 0.000 claims abstract description 10
- 238000000034 method Methods 0.000 claims description 17
- 230000005855 radiation Effects 0.000 claims description 15
- 238000005183 dynamical system Methods 0.000 claims description 14
- 238000012795 verification Methods 0.000 claims description 13
- 238000005516 engineering process Methods 0.000 claims description 7
- 230000008569 process Effects 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 5
- 230000001052 transient effect Effects 0.000 claims description 3
- 238000004422 calculation algorithm Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 11
- 238000004891 communication Methods 0.000 description 8
- 241000533950 Leucojum Species 0.000 description 5
- 241001269238 Data Species 0.000 description 4
- 238000007689 inspection Methods 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 238000004088 simulation Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011056 performance test Methods 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 238000004886 process control Methods 0.000 description 1
- 238000004153 renaturation Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The embodiment of the present invention provides a kind of operation data processing method and processing device of ship power system, which comprises the ship type for determining target ship power system obtains corresponding operation data according to the ship type and parses template；The listed files in the target ship power system equipment journal file folder is obtained, for any file in the listed files, the operation data record in the file is parsed using multithreading using operation data parsing template；After the completion of the document analysis, validity check and iterative testing are carried out to the operation data record after parsing, and corresponding all record in batch importing database by the operation data of validity check and iterative testing of the file is stored.The embodiment of the present invention, which realizes, reliably to be automatically parsed the operation data of ship power system and automated import of data library, and the efficiency analyzed the operation data of ship power system can be effectively improved.
Description
Technical field
The present invention relates to ship power system technical fields, more particularly, to a kind of operation number of ship power system
According to processing method and processing device.
Background technique
The operation data of ship power system (is hereinafter referred to as deposited mainly from ship power system operating parameter storage device
Storage device) record (being saved by binary file form), fraction carrys out the manual record of self-operating operator.Every is deposited
Storage device all corresponds to a set of power device, and automatically records its operating parameter.Data collecting instrument acquires power, electric power, radiates and prevent
The operation data for protecting each system is converted into normal data packet according to the transmission standard of particular communication agreement, by the number of each system
A complete record is just formed according to packet split, and is saved in storage device in the form of binary file.
Since power device operation data type is more (being divided into dynamical system, electric system, radiation protecting systems three parts),
Data volume is big (every set power device has a operating parameter more than 900, amounts at least 20 sets of power device data), available data file
The storage mode of form is not easy to statistically analyze.It is therefore desirable to, data volumes more to type greatly and in the form of binary file
The operation data of storage carries out automatic processing, these operating parameters are reliably automatically parsed and stored, so as at any time
Calling is checked historical data and is analyzed under emergency conditions data, and failure cause is traced.
Summary of the invention
The embodiment of the present invention provides a kind of ship power for overcoming the above problem or at least being partially solved the above problem
The operation data processing method and processing device of system.
In a first aspect, the embodiment of the present invention provides a kind of operation data processing method of ship power system, comprising:
The ship type for determining target ship power system obtains corresponding operation data according to the ship type and parses template；
The listed files in the target ship power system equipment journal file folder is obtained, in the listed files
Any file, using the operation data parsing template using multithreading in the file operation data record solve
Analysis；
After the completion of the document analysis, validity check is carried out to the operation data record after parsing and repeatability is examined
It looks into, and corresponding all recorded by the operation data of validity check and iterative testing of the file is imported into data in batches
It is stored in library；
Wherein, the operation data parsing template includes: offset, byte number, the type, classification reconciliation of field to be resolved
Analysis method.
Second aspect, the embodiment of the present invention provide a kind of operation data processing unit of ship power system, comprising:
It parses template and obtains module, for determining the ship type of target ship power system, obtained according to the ship type corresponding
Operation data parse template；
Parsing module, for obtaining the listed files in the target ship power system equipment journal file folder, for
Any file in the listed files, using operation data parsing template using multithreading to the operation in the file
Data record is parsed
Memory module, for carrying out validity school to the operation data record after parsing after the completion of the document analysis
It tests and iterative testing, and the corresponding all operation datas of validity check and iterative testing that pass through of the file is recorded
Batch is imported in database and is stored；
Wherein, the operation data parsing template includes: offset, byte number, the type, classification reconciliation of field to be resolved
Analysis method.
The third aspect, the embodiment of the present invention provides a kind of electronic equipment, including memory, processor and is stored in memory
Computer program that is upper and can running on a processor, is realized when the processor executes described program as first aspect provides
Ship power system operation data processing method the step of.
Fourth aspect, the embodiment of the present invention provide a kind of non-transient computer readable storage medium, are stored thereon with calculating
Machine program realizes the operation data of the ship power system as provided by first aspect when the computer program is executed by processor
The step of processing method.
The operation data processing method and processing device of ship power system provided in an embodiment of the present invention realizes dynamic to ship
The operation data of Force system reliably automatically parsed with automated import of data library, can effectively improve to ship power system
The efficiency analyzed of operation data.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is this hair
Bright some embodiments for those of ordinary skill in the art without creative efforts, can be with root
Other attached drawings are obtained according to these attached drawings.
Fig. 1 is the flow diagram of the operation data processing method of ship power system provided in an embodiment of the present invention；
Fig. 2 is that ship power system operation data acquires schematic diagram；
Fig. 3 is the resolving schematic diagram of worker thread provided in an embodiment of the present invention；
Fig. 4 is CRC check process schematic provided in an embodiment of the present invention；
Fig. 5 is the flow diagram of data check provided in an embodiment of the present invention；
Fig. 6 is the schematic diagram of the snowflake schema provided in an embodiment of the present invention for inquiring true table and dimension table composition；
Fig. 7 is the schematic diagram for the snowflake schema that statistical fact table provided in an embodiment of the present invention and dimension table form；
Fig. 8 is the structural schematic diagram of the operation data processing unit of ship power system provided in an embodiment of the present invention；
Fig. 9 is the entity structure schematic diagram of electronic equipment provided in an embodiment of the present invention.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, those of ordinary skill in the art
Every other embodiment obtained without creative efforts, shall fall within the protection scope of the present invention.
As shown in Figure 1, being a kind of stream of the operation data processing method of ship power system provided in an embodiment of the present invention
Journey schematic diagram, comprising:
Step 100, the ship type for determining target ship power system obtain corresponding operation data according to the ship type and parse
Template；
Specifically, as shown in Fig. 2, data collecting instrument acquisition power, electric power, each system of radiation protection operation data, root
It is converted into normal data packet according to the transmission standard of particular communication agreement, the data packet split of each system is just formed one completely
Record, and be saved in storage device, the data format of a complete documentation is as shown in table 1.
The data packet storing data format of each system is as shown in table 2 to table 4 in memory means record, electric system and radiation
Protection protection data packet has beginning and end flag bit, first part of the dynamical system as log, in data packet tail
Have check code, whether identification record effective, and the data packet of each system has corresponding date-time, if checksum test without
Accidentally, then the date-time recorded is subject to the date-time in dynamical system data packet.
The data format that table 1 records
2 dynamical system data packet storage format of table
3 electric power system data packet storage format of table
4 radiation protecting systems data packet storage format of table
The characteristics of in view of recording in storage device when carrying out data parsing is solved respectively by calculating offset
Analyse each field value.Offset is initialized as 0, every to have parsed a field value, is increased by offset, increased byte number depends on
The byte number shared by the field value parsed.If (counting date, time, flag bit etc. in by all fields and amounting to 976 words
Section) input of the byte number as analytical algorithm shared by offset or field in record, by largely inputting parameter ability
The data parsing algorithms of execution then lose the meaning of datamation, and if the data packet storage format of system occurs
Change, such as: new field is added in spare part, or modifies the byte number of existing field, and analytical algorithm will be again
Adjustment input parameter.
Therefore, the embodiment of the present invention passes through the characteristics of operation data for analyzing the target ship power system of different ship types,
Define corresponding operation data parsing template.The operation data parsing template includes: the offset of field to be resolved, byte
Number, type, classification and analytic method.Template is advantageous in that: 1) reusable: as long as the storage format of the record of parsing is not
It changes, then only needing to define once, can reuse always；2) be easy to extend: storage records data format
After change, it will be appropriately modified, can reuse after template duplicating；3) it is easy maintenance: after user modifies field attribute, by
In all field Coutinuous stores, template itself can automatically update other fields by information such as offset, the byte numbers of field
Information；4) adaptive good: can be according to the field information in template in resolving: such as field type and field classification, choosing
Select the corresponding analytical algorithm of field.Since the degree of coupling reduces, the addition and modification of analytical algorithm are also very convenient.By what is defined
Template is saved in database, is read corresponding template from database when parsing data every time and is parsed, can be in maximum
In degree, artificial intervention is reduced, truly realizes the automation of data processing.
Listed files in step 101, the acquisition target ship power system equipment journal file folder, for the text
Any file in part list remembers the operation data in the file using multithreading using operation data parsing template
Record is parsed；
Specifically, multithreading refers to the technology for realizing that multiple threads are concurrently executed from software or hardware.Due to each
It is big (up to 3 months logs, just there are 86400 records to generate daily, the nearly millions of total amount of data) to import data volume, it is single
Thread, which executes importing task, has that speed is slow, low efficiency.Therefore, the embodiment of the present invention improves number using multiple threads
According to the speed of the task of importing, original storage device record is divided by file, i.e. acquisition ship power system equipment journal file
Listed files in folder assigns files to several worker threads and is transported for any file in the listed files
The parsing of row data record also improves working efficiency while improving computer resource utilization rate.
For convenience of multithreading realization, 86400 (i.e. one day log) records are tested, each file is most
1000 records are stored, in total 87 files.Storage mode after being parsed according to operation data tests several think ofs of multithreading
Road is as follows:
(1) most common mode --- directly use thread pool.Per thread all completely parses file-storage at
Reason；Committed memory 1.2G or so.Thread pool size is 20, the used time 490 seconds.Compared to single thread processing, optimize about 200- on the time
300 seconds.It but is multithreading when due to storage, there are hidden danger.
(2) multithreading parsing is carried out to file, storage uses single thread.Specifically it is allocated as follows:
The parsing of 10 threads the used time 601 seconds, optimizes about 200 seconds or so, EMS memory occupation 1.5G；
The parsing of 20 threads, the used time 570 seconds, but EMS memory occupation reaches 1.95G；
Whole threads parse, after EMS memory occupation 13G, program crashing；
(3) multithreading parses, but only handles a file every time, single thread storage.A text first is parsed with multithreading
Record in part, then single storage；Used time 500 seconds, committed memory 190M or so.
The specific performance of 3 kinds of testing schemes is more as shown in table 5.The workflow of worker thread is as shown in Figure 3.Due to surveying
Examination data are that the analogue data using Program Generating is separated into integer and decimal using character string to guarantee the accurate of floating number
The strategy that two parts store respectively, it is longer that this directly results in calculation amount increase multithreading parsing.And under actual conditions, directly will be
Operation data is by numerical value storage, so parsing time-consuming will be reduced to the order of magnitude；For the analogue data generated with multithreading,
Time-consuming reduces about 40-50%；And parsing time-consuming can also drop to 500 seconds or so at present from 800+ seconds.
In view of data be written, the present invention use a file per treatment, multithreading solution new record, single thread storage
Mode.The record in real world memory devices is parsed using the program.500 records were about 1.4-1.8 seconds time-consuming, by 2 seconds
It calculates, one day 86400 record about 346 seconds.
5 testing scheme performance test of table
Wherein, described that the operation data in the file is recorded using multithreading using operation data parsing template
The step of being parsed, specifically:
The file is distributed into several worker threads, based on field to be resolved in operation data parsing template
Offset and analytic method, using several described worker threads parallel in the file operation data record by field into
Row parsing.
Specifically, a file is distributed into several worker threads, is parsed in template based on the operation data wait solve
The offset and the corresponding analytic method of each field to be resolved for analysing field, using several described threads and in a file
All operation datas record parsed by field.
Step 102, after the completion of the document analysis, validity check and again is carried out to the operation data record after parsing
Renaturation inspection, and corresponding all recorded by the operation data of validity check and iterative testing of the file is led in batches
Enter in database and is stored.
Specifically, after all operation datas record in one file is parsed, remember for each operation data
Record, check every record verification and, then calculate hash value to determine whether repetition record, if verification and correct and hash value without
Repetition will then know that this record can be written into database, and otherwise misregistration reason, carries out rollback processing, and parsing report is written
It accuses.
Corresponding all recorded by the operation data of validity check and iterative testing of one file is imported in batches
It is stored in database, realizes the single storage mode for a file.
The operation data processing method of ship power system provided in an embodiment of the present invention, realizes to ship power system
Operation data reliably automatically parsed with automated import of data library, the operation to ship power system can be effectively improved
The efficiency that data are analyzed.
Content based on the above embodiment, it is true according to the ship type in the ship type of the determining target ship power system
Before the step of determining operation data parsing template, further includes:
Operation data parsing template is defined according to the characteristics of different ship types；
Described the step of operation data parsing template is defined according to the characteristics of different ship types, specifically:
Define side of a ship table and flight number table；
Define basic data part.
Specifically, demand when importing for data, the corresponding different ship type of the side of a ship number, flight number correspond to the time that ship goes to sea
Section.Its dynamical system of different ship types has different design and realization, is embodied in subsystem difference, the subsystem operation of division
The number of parameter is different, operating parameter said system is different, and the equipment difference due to constituting dynamical system will also result in storage dress
When setting record operating parameter, the data type different (causing the length occupied in record not identical) of operating parameter and operation
The attributes such as the range bound of parameter, alarming value difference.So define ship type first, according to ship type the characteristics of, which formulates, to be corresponded to
Log parse template.Table 6 and table 7 are respectively the design of side of a ship table and flight number table.
6 side of a ship table of table
7 flight number table of table
Wherein, the step of definition basic data part, specifically:
Determine the essential information of each operating parameter of each subsystem of the ship power system of different ship types, the basic letter
Breath includes: ship type belonging to each operating parameter, affiliated subsystem, title, type, length and in memory means record
Offset；
A serial number is distributed to each operating parameter of each subsystem, and according to the type of operating parameter and affiliated subsystem
System, inputs other corresponding relevant informations of each operating parameter.
Specifically, basic data part refers to type, field generic, the field self attributes value etc. for constituting field,
Such as field type has the acquisition mould of date-time amount, amount of logic, the integer quantity of 4 bytes, the common simulation amount of 3 bytes, 2 bytes
Analog quantity and 4 byte analog quantitys (radiation protecting systems and electric system), these field types are directly determined in document analysis when institute
The data-bias of reading influences the accuracy of data parsing.Field generic has the field for belonging to dynamical system, radiation protection
System field and electric system field, each field are all based in the offset section of respective classification and are parsed.
In view of the flexibility of system, the balanced of big data quantity divides and the needs of search efficiency, and is directed to the later period
The actual needs such as more ships, more flight numbers is supported in extension, needs the operation such as creation physical table when importing, therefore basic data
Be based partially on field type (fieldtype), field generic (fieldcat), field attribute (field) three classes respectively into
Row maintenance, and data are generated based on this and parse template, related table structure is as shown in table 8 to table 10.
Since memory means record is segmented operating parameter according to the affiliated subsystem of operating parameter, same subsystem phase
The parameter of same type is continuously stored in record.Parse template defines each son of ship power system it needs to be determined that different ship types
The essential information of each operating parameter of system, the essential information include: ship type belonging to each operating parameter, affiliated son
System, title, type, length and the offset in memory means record；
Then, to one serial number of parametric distribution in each subsystem, in order to pass through sequence number search operating parameter later,
Then according to the type of operating parameter and affiliated subsystem, other corresponding relevant informations of operating parameter are inputted, such as analog quantity
Meaning under range bound, alarming value and unit, the default conditions of amount of logic and 01 state.
Table 8 joins field type table
Common field has date-time, amount of logic, device travel value, common simulation amount (3 byte), and power acquires analog quantity
(2 byte), radiation protection, power system simulation amount (4 byte).
9 field category table of table
10 Field Template table of table
Content based on the above embodiment, the operation data record after described pair of parsing carry out validity check and repeatability
The step of inspection, specifically:
Calculate verification that every operation data after parsing records and and hash value, judge verification and whether correct, and
Whether hash value is without repetition, if verification and correct and hash value know that this operation data records and pass through validity without repetition
Verification and iterative testing.
Specifically, the data packet group by not homologous ray is recorded in original storage device at for the reliability for guaranteeing data, solution
Common several data packet methods of calibration are provided in analysis algorithm.Most-often used is cyclic redundancy check.
Checking procedure and resolving:
Firstly, by the CRC code of storage device generation divided by the corresponding generated code of generator polynomial, if remainder is 0, CRC
Code is correct, and CRC check process is as shown in Figure 4.
Then, actual operation parameters numerical value is obtained.
Because of generated code R=3, redundant code is latter 3 i.e. 011, and actual information code is 1010.
Data record after parsing will finally be saved into database, duplicatedly record in verification and inspection in order to avoid importing
After errorless, in one embodiment, cryptographic Hash is generated using MD5 algorithm for every record, guarantee records without repetition and imports number
According in library.
The principle of Data Detection based on MD5 algorithm is as follows:
Input: the record date time, each system parameter analytic value
Output: 128 hash values
According to document, MD5 algorithm to be made to generate conflict, that is, input two different records and but export the same hash value,
Approximately 4,000,000,000 records generate the probability for repeating record when formula (1) indicates that record sum is n.
In formula, PrpIt indicates to generate the probability to conflict；N indicates the quantity of record；R indicates that optional two hash values are duplicate general
Rate, since hash value has 128, so r=2E-128.
N=1E7 is substituted into formula 2-1, it is P that the record for obtaining ten million magnitude, which generates the probability of Hash conflict,rp=1.47E-
25.The probability that Hash conflict occurs in view of MD5 algorithm is extremely low, it is believed that inputting different records in data importing process all will be defeated
A unique hash value out.
To sum up, the process of data check records as shown in figure 5, first reading from file, first checks the verification of every record
With, then hash value is calculated to determine whether repetition record, by record write-in data if verification and correct and hash value are without repeating
Library, otherwise misregistration reason, and parsing report is written.
Content based on the above embodiment, it is described to examine corresponding all validity check and the repeatability of passing through of the file
The operation data record batch looked into imports in database the step of being stored, specifically:
By the file it is corresponding it is all by the operation data of validity check and iterative testing record according to power
System, radiation protecting systems and electric system carry out a point table and store, and generate log basic information table.
Specifically, log basic information table mainly stores complete parsing log, i.e., the complete note parsed from file
Record content.For Given information, field number is up to 995 or more in original record, in addition necessary control field, has exceeded
Limitation of the relevant database of mainstream to field in the market, therefore, this part use by dynamical system, radiation protecting systems and
Electric system carries out a point table and stores；Simultaneously in view of the big problem of data volume, in conjunction with bottom data cluster-based storage characteristic, using pressing
Equipment-monthly divides table to store, and guarantees single table data record amount (about 3600*24*30=2,592,000) within 3,000,000, mentions
Height storage and recall precision.
It in view of dynamical system, radiation protecting systems and electric system three classes literary name section are numerous, and is compatible different ship
Only, different-format, manual maintenance or table structure template, task amount are huge.Therefore, table creation task is transferred into storing process control
System automatically generates, and the following are the table structure examples of creation, as shown in table 11 to table 13.
11 dynamical system table of table
12 radiation protecting systems table of table
13 electric system table of table
There are also the operating parameter tables that user manually enters, and it is as shown in table 14 to establish exemplary construction.
The manually recorded table of table 14
In view of that there are types is more for manually recorded log, the features such as frequency is low, selection uses the side of Excel table
Formula imports, and table structure stays spare there are three extended field, can be used for storing and uploads document location, analysis state etc..
In fact, log derivation information table can also be generated on the basis of log basic information table.Main purpose in order to
Data query and analysis efficiency are improved, mainly includes data summary table, concordance list, user query template.
According to current needs, storage system has designed partial data statistic summary table, such as: daily, control week, monthly count
Equipment situation calculates and stores the facility information of this day, calculates the arithmetic equal value, standard variance, maximum value, minimum of each field
It is worth, the points that peel off, more dividing value etc., fast and easy inquiry and abnormal positioning.
According to it is above-mentioned configured template after, each operating parameter is checked according to the attribute of operating parameter according to certain algorithm
Whether byte offsets and field are correct by bit length in record, when user modifies the category of the operating parameter in parsing template
(4 byte analog quantity of the position int32 is such as changed to 3 byte analog quantity of the position int24) when property, can automaticly inspect and correct.
Byte number shared by date-time offset+date-time=dynamical system offset
Byte number=radiation protecting systems offset shared by dynamical system offset+each power system parameter * parameter
Byte number shared by radiation protecting systems offset+each radiation protecting systems parameter * parameter=electric system offset
Amount.
Dynamical system offset value is start-stop position of the power system parameter in a log, analytical algorithm root
The parsing of different modes is executed to different system parameter according to offset.To guarantee data importing process controllable precise, can also give birth to
At data parsing template and parsing log sheet, dependency structure is as shown in Table 15 and Table 16.
Table 15 parses template table
Table 16 parses log sheet
Content based on the above embodiment, after the generation log basic information table, further includes: using at on-line analysis
OLAP technology is managed, on the basis of the log basic information table, constructs true table and dimension table.
Specifically, on-line analytical processing (Online Analytical Processing, OLAP technology) is to what is analyzed
Each attribute that the problem of problem is abstracted, and will be analyzed shows is regarded as dimension.A kind of dimension correspondence problem is shown
An attribute, dimension can also be divided into many levels according to the characteristics of this attribute.So it is directed to particular problem,
Can multiple and different dimensions from problem and different levels, problem is investigated comprehensively, or for certain in problem
One attribute paid close attention to, carries out careful analysis.After defining multiple dimensions, can also according to it is to be analyzed the problem of spy
Defined good different dimensions are combined by point, and being formed with problem is the fact that be oriented to table.Generally according to a true table
With multiple dimension tables, so that it may form Star Model and snowflake type model both storage models, both models also precisely with
The summary of problem after abstract.
It inquires true table and statistical fact table is the result table generated after data aggregate according to some dimension.Inquire true table
With the schematic diagram of the snowflake schema of dimension table composition as shown in fig. 6, showing for the snowflake schema of statistical fact table and dimension table composition
It is intended to as shown in Figure 7.Wherein, query time, screening frequency are as time dimension table, and clock rate, parameter type are as parameter
Dimension table, query template, the side of a ship number, the flight number condition used as inquiry.Wherein, statistical parameter is analog quantity parameter, statistics fortune
Row parameter is in the maximum value of certain time period, minimum value, average value and parameter-beyond-limit or causes the number alarmed.
As shown in figure 8, the structure for the operation data processing unit of ship power system provided in an embodiment of the present invention is shown
It is intended to, comprising: parsing template obtains module 801, parsing module 802 and memory module 803, wherein
It parses template and obtains module 801, for determining the ship type of target ship power system, phase is obtained according to the ship type
The operation data parsing template answered；
Parsing module 802, for for obtaining the file column in the target ship power system equipment journal file folder
Table, for any file in the listed files, using operation data parsing template using multithreading to the file
In operation data record parsed
Memory module 803, for carrying out validity to the operation data record after parsing after the completion of the document analysis
Verification and iterative testing, and by the file it is corresponding it is all pass through the operation datas of validity check and iterative testing remember
Record batch, which imports in database, to be stored；
Wherein, the operation data parsing template includes: offset, byte number, the type, classification reconciliation of field to be resolved
Analysis method.
The operation data processing unit of ship power system is used to execute the ship power system, area in above method embodiment
The operation data processing method of system.Therefore, in the operation data processing method of the ship power system in foregoing embodiments
Description and definition, can be used for the understanding of the operation data processing unit of ship power system in the embodiment of the present invention, herein
It repeats no more.
Fig. 9 is the entity structure schematic diagram of electronic equipment provided in an embodiment of the present invention, as shown in figure 9, the electronic equipment
It may include: processor (processor) 910,920, memory communication interface (Communications Interface)
(memory) 930 and communication bus 940, wherein processor 910, communication interface 920, memory 930 pass through communication bus 940
Complete mutual communication.Processor 910 can call the meter that is stored on memory 930 and can run on processor 910
Calculation machine program to execute the operation data processing method of ship power system provided by above-mentioned each method embodiment, such as is wrapped
It includes: determining the ship type of target ship power system, corresponding operation data is obtained according to the ship type and parses template；Described in acquisition
Listed files in target ship power system equipment journal file folder is utilized for any file in the listed files
The operation data parsing template parses the operation data record in the file using multithreading；In the file solution
After the completion of analysis, validity check and iterative testing are carried out to the operation data record after parsing, and the file is corresponding
All recorded in batch importing database by the operation data of validity check and iterative testing is stored；Wherein, institute
State offset, byte number, type, classification and the analytic method that operation data parsing template includes: field to be resolved.
In addition, the logical order in above-mentioned memory 930 can be realized by way of SFU software functional unit and conduct
Independent product when selling or using, can store in a computer readable storage medium.Based on this understanding, originally
The technical solution of the inventive embodiments substantially part of the part that contributes to existing technology or the technical solution in other words
It can be expressed in the form of software products, which is stored in a storage medium, including some instructions
With so that computer equipment (can be personal computer, server or the network equipment an etc.) execution present invention is each
The all or part of the steps of embodiment the method.And storage medium above-mentioned includes: USB flash disk, mobile hard disk, read-only memory
(ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic or disk
Etc. the various media that can store program code.
The embodiment of the present invention also provides a kind of non-transient computer readable storage medium, is stored thereon with computer program,
The computer program is realized when being executed by processor at the operation data for the ship power system that above-mentioned each method embodiment provides
Reason method, for example, the ship type for determining target ship power system obtains corresponding operation data according to the ship type and parses
Template；The listed files in the target ship power system equipment journal file folder is obtained, in the listed files
Any file solves the operation data record in the file using multithreading using operation data parsing template
Analysis；After the completion of the document analysis, validity check and iterative testing are carried out to the operation data record after parsing, and will
The file it is corresponding it is all by the operation data of validity check and iterative testing record batch import in database into
Row storage；Wherein, the operation data parsing template includes: offset, byte number, the type, classification reconciliation of field to be resolved
Analysis method.
The apparatus embodiments described above are merely exemplary, wherein described, unit can as illustrated by the separation member
It is physically separated with being or may not be, component shown as a unit may or may not be physics list
Member, it can it is in one place, or may be distributed over multiple network units.It can be selected according to the actual needs
In some or all of the modules achieve the purpose of the solution of this embodiment.Those of ordinary skill in the art are not paying creativeness
Labour in the case where, it can understand and implement.
Through the above description of the embodiments, those skilled in the art can be understood that each embodiment can
It realizes by means of software and necessary general hardware platform, naturally it is also possible to pass through hardware.Based on this understanding, on
Stating technical solution, substantially the part that contributes to existing technology can be embodied in the form of software products in other words, should
Computer software product may be stored in a computer readable storage medium, such as ROM/RAM, magnetic disk, CD, including several fingers
It enables and using so that a computer equipment (can be personal computer, server or the network equipment etc.) executes each implementation
Method described in certain parts of example or embodiment.
Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of the present invention, rather than its limitations；Although
Present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that: it still may be used
To modify the technical solutions described in the foregoing embodiments or equivalent replacement of some of the technical features；
And these are modified or replaceed, technical solution of various embodiments of the present invention that it does not separate the essence of the corresponding technical solution spirit and
Range.
Claims (10)
1. a kind of operation data processing method of ship power system characterized by comprising
The ship type for determining target ship power system obtains corresponding operation data according to the ship type and parses template；
The listed files in the target ship power system equipment journal file folder is obtained, for appointing in the listed files
One file parses the operation data record in the file using multithreading using operation data parsing template；
After the completion of the document analysis, validity check and iterative testing are carried out to the operation data record after parsing, and
Corresponding all recorded by the operation data of validity check and iterative testing of the file is imported in database in batches
It is stored；
Wherein, the operation data parsing template includes: offset, byte number, type, classification and the parsing side of field to be resolved
Method.
2. the operation data processing method of ship power system according to claim 1, which is characterized in that in the determination
The ship type of target ship power system, according to the ship type determine operation data parse template the step of before, further includes:
Operation data parsing template is defined according to the characteristics of different ship types；
Described the step of operation data parsing template is defined according to the characteristics of different ship types, specifically:
Define side of a ship table and flight number table；
Define basic data part.
3. the operation data processing method of ship power system according to claim 2, which is characterized in that the definition base
The step of plinth data portion, specifically:
Determine the essential information of each operating parameter of each subsystem of the ship power system of different ship types, the essential information packet
It includes: ship type belonging to each operating parameter, affiliated subsystem, title, type, length and the offset in memory means record
Amount；
A serial number is distributed to each operating parameter of each subsystem, and according to the type of operating parameter and affiliated subsystem,
Input other corresponding relevant informations of each operating parameter.
4. the operation data processing method of ship power system according to claim 1, which is characterized in that described to utilize institute
The step of operation data parsing template parses the operation data record in the file using multithreading is stated, specifically:
The file is distributed into several worker threads, the offset based on field to be resolved in operation data parsing template
Amount and analytic method parallel record the operation data in the file using several described worker threads and solve by field
Analysis.
5. the operation data processing method of ship power system according to claim 1, which is characterized in that described pair of parsing
The step of operation data record afterwards carries out validity check and iterative testing, specifically:
Calculate verification that every operation data after parsing records and and hash value, judge verification and whether correct and Hash
It is worth whether without repetition, if verification and correct and hash value know that this operation data records and pass through validity check without repetition
And iterative testing.
6. the operation data processing method of ship power system according to claim 1, which is characterized in that it is described will be described
Corresponding all recorded in batch importing database by the operation data of validity check and iterative testing of file is deposited
The step of storage, specifically:
By the file it is corresponding it is all by the operation data of validity check and iterative testing record according to dynamical system,
Radiation protecting systems and electric system carry out a point table and store, and generate log basic information table.
7. the operation data processing method of ship power system according to claim 6, which is characterized in that the generation day
After will basic information table, further includes:
Using On Line Analysis Process technology, on the basis of the log basic information table, true table and dimension table are constructed.
8. a kind of operation data processing unit of ship power system characterized by comprising
It parses template and obtains module, for determining the ship type of target ship power system, corresponding fortune is obtained according to the ship type
Row data parse template；
Parsing module, for obtaining the listed files in the target ship power system equipment journal file folder, for described
Any file in listed files, using operation data parsing template using multithreading to the operation data in the file
Record is parsed
Memory module, for after the completion of the document analysis, to the operation data record after parsing carry out validity check and
Iterative testing, and by the file it is corresponding it is all pass through the operation data of validity check and iterative testing record batch
It imports in database and is stored；
Wherein, the operation data parsing template includes: offset, byte number, type, classification and the parsing side of field to be resolved
Method.
9. a kind of electronic equipment including memory, processor and stores the calculating that can be run on a memory and on a processor
Machine program, which is characterized in that the processor realizes that ship is dynamic as described in any one of claim 1 to 7 when executing described program
The step of operation data processing method of Force system.
10. a kind of non-transient computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer
The operation data processing method of the ship power system as described in any one of claim 1 to 7 is realized when program is executed by processor
The step of.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910681803.0A CN110532612A (en) | 2019-07-26 | 2019-07-26 | The operation data processing method and processing device of ship power system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910681803.0A CN110532612A (en) | 2019-07-26 | 2019-07-26 | The operation data processing method and processing device of ship power system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110532612A true CN110532612A (en) | 2019-12-03 |
Family
ID=68661946
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910681803.0A Pending CN110532612A (en) | 2019-07-26 | 2019-07-26 | The operation data processing method and processing device of ship power system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110532612A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111597195A (en) * | 2020-05-07 | 2020-08-28 | 交通运输部水运科学研究所 | Ship atmospheric pollution emission list missing information processing method and device |
| CN113672569A (en) * | 2021-09-01 | 2021-11-19 | 封�波 | Method and device for automatically recording electronic navigation log |
| CN114780497A (en) * | 2022-04-22 | 2022-07-22 | 湖南长银五八消费金融股份有限公司 | Batch file processing method, apparatus, computer device, medium, and program product |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101458705A (en) * | 2008-12-29 | 2009-06-17 | 阿里巴巴集团控股有限公司 | Data collating method between different utility systems, apparatus and system |
| CN102567401A (en) * | 2010-12-31 | 2012-07-11 | 中国科学院沈阳自动化研究所 | Manned submersible data analysis system |
| CN103135515A (en) * | 2011-12-05 | 2013-06-05 | 北京掌城科技有限公司 | Diagnostic method for vehicle condition |
| CN106156170A (en) * | 2015-04-16 | 2016-11-23 | 北大方正集团有限公司 | The analysis of public opinion method and device |
| CN107122434A (en) * | 2017-04-18 | 2017-09-01 | 北京思特奇信息技术股份有限公司 | A kind of method and system that reconciliation file is imported to database |
| CN107945090A (en) * | 2017-11-30 | 2018-04-20 | 深圳市轱辘车联数据技术有限公司 | Vehicular exhaust data analysing method, device and server based on block chain |
| US20180284737A1 (en) * | 2016-05-09 | 2018-10-04 | StrongForce IoT Portfolio 2016, LLC | Methods and systems for detection in an industrial internet of things data collection environment with large data sets |
| CN108880867A (en) * | 2018-05-31 | 2018-11-23 | 腾讯科技（深圳）有限公司 | A kind of network equipment collecting method and system |
| CN109711682A (en) * | 2018-12-13 | 2019-05-03 | 中国船舶工业***工程研究院 | A kind of ship overall status appraisal procedure of data-driven |
-
2019
- 2019-07-26 CN CN201910681803.0A patent/CN110532612A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101458705A (en) * | 2008-12-29 | 2009-06-17 | 阿里巴巴集团控股有限公司 | Data collating method between different utility systems, apparatus and system |
| CN102567401A (en) * | 2010-12-31 | 2012-07-11 | 中国科学院沈阳自动化研究所 | Manned submersible data analysis system |
| CN103135515A (en) * | 2011-12-05 | 2013-06-05 | 北京掌城科技有限公司 | Diagnostic method for vehicle condition |
| CN106156170A (en) * | 2015-04-16 | 2016-11-23 | 北大方正集团有限公司 | The analysis of public opinion method and device |
| US20180284737A1 (en) * | 2016-05-09 | 2018-10-04 | StrongForce IoT Portfolio 2016, LLC | Methods and systems for detection in an industrial internet of things data collection environment with large data sets |
| CN107122434A (en) * | 2017-04-18 | 2017-09-01 | 北京思特奇信息技术股份有限公司 | A kind of method and system that reconciliation file is imported to database |
| CN107945090A (en) * | 2017-11-30 | 2018-04-20 | 深圳市轱辘车联数据技术有限公司 | Vehicular exhaust data analysing method, device and server based on block chain |
| CN108880867A (en) * | 2018-05-31 | 2018-11-23 | 腾讯科技（深圳）有限公司 | A kind of network equipment collecting method and system |
| CN109711682A (en) * | 2018-12-13 | 2019-05-03 | 中国船舶工业***工程研究院 | A kind of ship overall status appraisal procedure of data-driven |
Non-Patent Citations (1)
| Title |
|---|
| 常杰: "复杂***的数据分析技术在动车组数字化调试线中的应用", 《计算机测量与控制》 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111597195A (en) * | 2020-05-07 | 2020-08-28 | 交通运输部水运科学研究所 | Ship atmospheric pollution emission list missing information processing method and device |
| CN111597195B (en) * | 2020-05-07 | 2023-03-14 | 交通运输部水运科学研究所 | Ship atmospheric pollution emission list missing information processing method and device |
| CN113672569A (en) * | 2021-09-01 | 2021-11-19 | 封�波 | Method and device for automatically recording electronic navigation log |
| CN113672569B (en) * | 2021-09-01 | 2024-05-07 | 上海太科舟智能科技有限公司 | Method and device for automatically recording electronic navigation log |
| CN114780497A (en) * | 2022-04-22 | 2022-07-22 | 湖南长银五八消费金融股份有限公司 | Batch file processing method, apparatus, computer device, medium, and program product |
| CN114780497B (en) * | 2022-04-22 | 2023-02-24 | 湖南长银五八消费金融股份有限公司 | Batch file processing method, apparatus, computer device, medium, and program product |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP4195112A1 (en) | Systems and methods for enriching modeling tools and infrastructure with semantics | |
| CN110532612A (en) | The operation data processing method and processing device of ship power system | |
| CN111563016B (en) | Log collection and analysis method and device, computer system and readable storage medium | |
| US20160092290A1 (en) | Processing data errors for a data processing system | |
| CN111562965B (en) | Page data verification method and device based on decision tree | |
| CN111259040B (en) | SQL statement auditing method and system | |
| US10394700B1 (en) | System and method for identifying a subset of total historical users of a data management system to represent a full set of test scenarios based on prehashing of code coverage information | |
| CN108460068A (en) | Method, apparatus, storage medium and the terminal that report imports and exports | |
| CN111949540A (en) | Code change risk estimation and verification method and device | |
| CN111737244A (en) | Data quality inspection method, device, computer system and storage medium | |
| CN112801315A (en) | State diagnosis method and device for power secondary equipment and terminal | |
| Jiang et al. | A Large-scale Benchmark for Log Parsing | |
| Baresi et al. | Test oracles for simulink-like models | |
| CN114926082B (en) | Data fluctuation early warning method based on artificial intelligence and related equipment | |
| Jiang et al. | A Large-Scale Evaluation for Log Parsing Techniques: How Far Are We? | |
| CN116860311A (en) | Script analysis method, script analysis device, computer equipment and storage medium | |
| Offutt et al. | An industrial study of applying input space partitioning to test financial calculation engines | |
| CN114595216A (en) | Data verification method and device, storage medium and electronic equipment | |
| CN114491044A (en) | Log processing method and device | |
| CN113919609A (en) | Power distribution network model quality comprehensive evaluation method and system | |
| CN117829121B (en) | Data processing method, device, electronic equipment and medium | |
| CN110399298A (en) | A kind of test method and device | |
| CN117076546B (en) | Data processing method, terminal device and computer readable storage medium | |
| CN111968022B (en) | Service number generation system and method based on JSON configuration mode | |
| CN110119406B (en) | Method and device for checking real-time task records |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191203 |