CN116910016B - AIS data processing method - Google Patents
AIS data processing method Download PDFInfo
- Publication number
- CN116910016B CN116910016B CN202311182087.4A CN202311182087A CN116910016B CN 116910016 B CN116910016 B CN 116910016B CN 202311182087 A CN202311182087 A CN 202311182087A CN 116910016 B CN116910016 B CN 116910016B
- Authority
- CN
- China
- Prior art keywords
- ais
- data
- disk
- hdfs
- kafka
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 18
- 238000012360 testing method Methods 0.000 claims abstract description 35
- 238000012545 processing Methods 0.000 claims abstract description 22
- 230000008859 change Effects 0.000 claims description 30
- 230000004927 fusion Effects 0.000 claims description 13
- 238000005457 optimization Methods 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 5
- 230000006870 function Effects 0.000 claims description 4
- 238000005192 partition Methods 0.000 claims description 4
- 230000002688 persistence Effects 0.000 claims description 4
- 230000004044 response Effects 0.000 claims description 4
- 230000005540 biological transmission Effects 0.000 abstract description 3
- 238000012544 monitoring process Methods 0.000 abstract 1
- 238000000034 method Methods 0.000 description 5
- 238000010276 construction Methods 0.000 description 4
- 238000007726 management method Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000004088 simulation Methods 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/119—Details of migration of file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24578—Query processing with adaptation to user needs using ranking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0688—Non-volatile semiconductor memory arrays
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0689—Disk arrays, e.g. RAID, JBOD
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Human Computer Interaction (AREA)
- Software Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Claims (5)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202311182087.4A CN116910016B (en) | 2023-09-14 | 2023-09-14 | AIS data processing method |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202311182087.4A CN116910016B (en) | 2023-09-14 | 2023-09-14 | AIS data processing method |
Publications (2)
Publication Number | Publication Date |
---|---|
CN116910016A CN116910016A (en) | 2023-10-20 |
CN116910016B true CN116910016B (en) | 2024-06-11 |
Family
ID=88367350
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202311182087.4A Active CN116910016B (en) | 2023-09-14 | 2023-09-14 | AIS data processing method |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN116910016B (en) |
Citations (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104583930A (en) * | 2014-08-15 | 2015-04-29 | 华为技术有限公司 | Method of data migration, controller and data migration apparatus |
CN104944240A (en) * | 2015-05-19 | 2015-09-30 | 重庆大学 | Elevator equipment state monitoring system based on large data technology |
CN106709003A (en) * | 2016-12-23 | 2017-05-24 | 长沙理工大学 | Hadoop-based mass log data processing method |
CN107491448A (en) * | 2016-06-12 | 2017-12-19 | ***通信集团四川有限公司 | A kind of HBase resource adjusting methods and device |
CN107644050A (en) * | 2016-12-22 | 2018-01-30 | 北京锐安科技有限公司 | A kind of querying method and device of the Hbase based on solr |
CN109525593A (en) * | 2018-12-20 | 2019-03-26 | 中科曙光国际信息产业有限公司 | A kind of pair of hadoop big data platform concentrates security management and control system and method |
CN110515726A (en) * | 2019-08-14 | 2019-11-29 | 苏州浪潮智能科技有限公司 | A kind of database loads equalization methods and device |
CN110865989A (en) * | 2019-11-22 | 2020-03-06 | 浪潮电子信息产业股份有限公司 | Business processing method for large-scale computing cluster |
CN111177271A (en) * | 2019-12-31 | 2020-05-19 | 奇安信科技集团股份有限公司 | Data storage method and device for persistence of kafka data to hdfs, and computer equipment |
CN111966656A (en) * | 2020-07-17 | 2020-11-20 | 苏州浪潮智能科技有限公司 | Method, system, terminal and storage medium for simulating high-load scene of storage file |
CN112433845A (en) * | 2020-10-29 | 2021-03-02 | 苏州浪潮智能科技有限公司 | HBase service management method, device, equipment and readable medium |
CN112799597A (en) * | 2021-02-08 | 2021-05-14 | 东北大学 | Hierarchical storage fault-tolerant method for stream data processing |
CN115934251A (en) * | 2022-12-06 | 2023-04-07 | 浪潮云信息技术股份公司 | Method and system for realizing high availability of cloud native NFS |
CN116383167A (en) * | 2022-12-27 | 2023-07-04 | 爱信诺征信有限公司 | Method for solving insufficient disk space based on object storage |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10642800B2 (en) * | 2013-10-25 | 2020-05-05 | Vmware, Inc. | Multi-tenant distributed computing and database |
US9990383B2 (en) * | 2015-05-01 | 2018-06-05 | Ebay Inc. | Constructing a data adaptor in an enterprise server data ingestion environment |
US20200341855A1 (en) * | 2019-04-28 | 2020-10-29 | Synamedia | Object store specialized backup and point-in-time recovery architecture |
-
2023
- 2023-09-14 CN CN202311182087.4A patent/CN116910016B/en active Active
Patent Citations (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104583930A (en) * | 2014-08-15 | 2015-04-29 | 华为技术有限公司 | Method of data migration, controller and data migration apparatus |
CN104944240A (en) * | 2015-05-19 | 2015-09-30 | 重庆大学 | Elevator equipment state monitoring system based on large data technology |
CN107491448A (en) * | 2016-06-12 | 2017-12-19 | ***通信集团四川有限公司 | A kind of HBase resource adjusting methods and device |
CN107644050A (en) * | 2016-12-22 | 2018-01-30 | 北京锐安科技有限公司 | A kind of querying method and device of the Hbase based on solr |
CN106709003A (en) * | 2016-12-23 | 2017-05-24 | 长沙理工大学 | Hadoop-based mass log data processing method |
CN109525593A (en) * | 2018-12-20 | 2019-03-26 | 中科曙光国际信息产业有限公司 | A kind of pair of hadoop big data platform concentrates security management and control system and method |
CN110515726A (en) * | 2019-08-14 | 2019-11-29 | 苏州浪潮智能科技有限公司 | A kind of database loads equalization methods and device |
CN110865989A (en) * | 2019-11-22 | 2020-03-06 | 浪潮电子信息产业股份有限公司 | Business processing method for large-scale computing cluster |
CN111177271A (en) * | 2019-12-31 | 2020-05-19 | 奇安信科技集团股份有限公司 | Data storage method and device for persistence of kafka data to hdfs, and computer equipment |
CN111966656A (en) * | 2020-07-17 | 2020-11-20 | 苏州浪潮智能科技有限公司 | Method, system, terminal and storage medium for simulating high-load scene of storage file |
CN112433845A (en) * | 2020-10-29 | 2021-03-02 | 苏州浪潮智能科技有限公司 | HBase service management method, device, equipment and readable medium |
CN112799597A (en) * | 2021-02-08 | 2021-05-14 | 东北大学 | Hierarchical storage fault-tolerant method for stream data processing |
CN115934251A (en) * | 2022-12-06 | 2023-04-07 | 浪潮云信息技术股份公司 | Method and system for realizing high availability of cloud native NFS |
CN116383167A (en) * | 2022-12-27 | 2023-07-04 | 爱信诺征信有限公司 | Method for solving insufficient disk space based on object storage |
Also Published As
Publication number | Publication date |
---|---|
CN116910016A (en) | 2023-10-20 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Lakshman et al. | Cassandra: a decentralized structured storage system | |
CN104735110B (en) | Metadata management method and system | |
CN107180113B (en) | Big data retrieval platform | |
Kaiser et al. | Design of an exact data deduplication cluster | |
Fu et al. | Performance optimization for managing massive numbers of small files in distributed file systems | |
CN109918450B (en) | Distributed parallel database based on analysis type scene and storage method | |
EP2502167A1 (en) | Super-records | |
Wang et al. | {MAPX}: Controlled data migration in the expansion of decentralized {Object-Based} storage systems | |
US20200192805A1 (en) | Adaptive Cache Commit Delay for Write Aggregation | |
CN106066890A (en) | A kind of distributed high-performance data storehouse integrated machine system | |
Otoo et al. | Disk cache replacement algorithm for storage resource managers in data grids | |
Garefalakis et al. | Acazoo: A distributed key-value store based on replicated lsm-trees | |
CN111813332A (en) | High-performance, high-expansion and high-safety intelligent distributed storage system | |
CN113032356B (en) | Cabin distributed file storage system and implementation method | |
CN109716280B (en) | Flexible memory rank storage arrangement | |
Qi et al. | Big data management in digital forensics | |
CN116910016B (en) | AIS data processing method | |
CN102867029B (en) | A kind of method managing distributive catalogue of document system and distributed file system | |
Rodriguez et al. | Unifying the data center caching layer: Feasible? profitable? | |
Ren et al. | Towards realistic benchmarking for cloud file systems: Early experiences | |
Xu et al. | TEA: A traffic-efficient erasure-coded archival scheme for in-memory stores | |
Xu et al. | Traffic-aware erasure-coded archival schemes for in-memory stores | |
Liu et al. | SSD as a Cloud Cache? Carefully Design about It | |
Shvidkiy et al. | Caching methods analysis for improving distributed storage systems performance | |
Ji et al. | Transaction data management system based on distributed storage architecture |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
CB03 | Change of inventor or designer information | ||
CB03 | Change of inventor or designer information |
Inventor after: Zhang Jiandong Inventor after: Mo Peipei Inventor after: Zhao Fenglong Inventor after: Yun Zeyu Inventor after: Hu Qing Inventor after: Ma Rong Inventor after: Li Yang Inventor after: Wang Zhenjiang Inventor after: Li Jianping Inventor after: Yin Fan Inventor before: Zhao Fenglong Inventor before: Mo Peipei Inventor before: Yun Zeyu Inventor before: Zhang Jiandong Inventor before: Hu Qing Inventor before: Ma Rong Inventor before: Li Yang Inventor before: Wang Zhenjiang Inventor before: Li Jianping Inventor before: Yin Fan |
|
GR01 | Patent grant | ||
GR01 | Patent grant |