CN112035057B - 一种hive文件合并的方法和装置 - Google Patents
一种hive文件合并的方法和装置 Download PDFInfo
- Publication number
- CN112035057B CN112035057B CN202010725466.3A CN202010725466A CN112035057B CN 112035057 B CN112035057 B CN 112035057B CN 202010725466 A CN202010725466 A CN 202010725466A CN 112035057 B CN112035057 B CN 112035057B
- Authority
- CN
- China
- Prior art keywords
- files
- file
- bucket
- merged
- merging
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1727—Details of free space management performed by the file system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
- G06F3/0631—Configuration or reconfiguration of storage systems by allocating resources to storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Claims (10)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010725466.3A CN112035057B (zh) | 2020-07-24 | 2020-07-24 | 一种hive文件合并的方法和装置 |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010725466.3A CN112035057B (zh) | 2020-07-24 | 2020-07-24 | 一种hive文件合并的方法和装置 |
Publications (2)
Publication Number | Publication Date |
---|---|
CN112035057A CN112035057A (zh) | 2020-12-04 |
CN112035057B true CN112035057B (zh) | 2022-06-21 |
Family
ID=73583098
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202010725466.3A Active CN112035057B (zh) | 2020-07-24 | 2020-07-24 | 一种hive文件合并的方法和装置 |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN112035057B (zh) |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103677664A (zh) * | 2012-09-04 | 2014-03-26 | 国际商业机器公司 | 用于按需缓存的方法和数据处理*** |
CN107861686A (zh) * | 2017-09-26 | 2018-03-30 | 深圳前海微众银行股份有限公司 | 文件存储方法、服务端和计算机可读存储介质 |
CN109063192A (zh) * | 2018-08-29 | 2018-12-21 | 广州洪荒智能科技有限公司 | 一种高性能海量文件存储***工作方法 |
CN109542682A (zh) * | 2018-11-16 | 2019-03-29 | 上海达梦数据库有限公司 | 一种数据备份方法、装置、设备和存储介质 |
CN109726177A (zh) * | 2018-12-29 | 2019-05-07 | 北京赛思信安技术股份有限公司 | 一种基于HBase的海量文件分区索引方法 |
US10360076B1 (en) * | 2016-10-06 | 2019-07-23 | Sprint Communications Company L.P. | Prioritized rebalancing of distributed file system |
CN110321329A (zh) * | 2019-06-18 | 2019-10-11 | 中盈优创资讯科技有限公司 | 基于大数据的数据处理方法及装置 |
CN111026768A (zh) * | 2019-10-16 | 2020-04-17 | 武汉达梦数据库有限公司 | 一种可实现数据快速装载的数据同步方法和装置 |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7996445B2 (en) * | 2007-04-27 | 2011-08-09 | Network Appliance, Inc. | Block reallocation planning during read-ahead processing |
-
2020
- 2020-07-24 CN CN202010725466.3A patent/CN112035057B/zh active Active
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103677664A (zh) * | 2012-09-04 | 2014-03-26 | 国际商业机器公司 | 用于按需缓存的方法和数据处理*** |
US10360076B1 (en) * | 2016-10-06 | 2019-07-23 | Sprint Communications Company L.P. | Prioritized rebalancing of distributed file system |
CN107861686A (zh) * | 2017-09-26 | 2018-03-30 | 深圳前海微众银行股份有限公司 | 文件存储方法、服务端和计算机可读存储介质 |
CN109063192A (zh) * | 2018-08-29 | 2018-12-21 | 广州洪荒智能科技有限公司 | 一种高性能海量文件存储***工作方法 |
CN109542682A (zh) * | 2018-11-16 | 2019-03-29 | 上海达梦数据库有限公司 | 一种数据备份方法、装置、设备和存储介质 |
CN109726177A (zh) * | 2018-12-29 | 2019-05-07 | 北京赛思信安技术股份有限公司 | 一种基于HBase的海量文件分区索引方法 |
CN110321329A (zh) * | 2019-06-18 | 2019-10-11 | 中盈优创资讯科技有限公司 | 基于大数据的数据处理方法及装置 |
CN111026768A (zh) * | 2019-10-16 | 2020-04-17 | 武汉达梦数据库有限公司 | 一种可实现数据快速装载的数据同步方法和装置 |
Non-Patent Citations (2)
Title |
---|
一种基于混合索引的HDFS小文件存储策略;熊安萍等;《重庆邮电大学学报(自然科学版)》;20150215(第01期);全文 * |
基于网络空间安全实时数据的HDFS小文件问题研究;王绍节等;《信息网络安全》;20171010(第10期);全文 * |
Also Published As
Publication number | Publication date |
---|---|
CN112035057A (zh) | 2020-12-04 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9575976B2 (en) | Methods and apparatuses to optimize updates in a file system based on birth time | |
US10831736B2 (en) | Fast multi-tier indexing supporting dynamic update | |
US9110826B2 (en) | Memory allocation in a system using memory striping | |
CN101447940B (zh) | 访问控制列表规则的更新方法和装置 | |
CN106407207B (zh) | 一种实时新增数据更新方法和装置 | |
CN110083601A (zh) | 面向键值存储***的索引树构建方法及*** | |
CN109522428B (zh) | 一种基于索引定位的图计算***的外存访问方法 | |
EP4105793A1 (en) | Signature-based cache optimization for data preparation | |
CN106095589A (zh) | 一种分配分区的方法、装置及*** | |
WO2013032436A1 (en) | Parallel operation on b+ trees | |
CN107992577A (zh) | 一种哈希表数据冲突处理方法及装置 | |
CN106599091A (zh) | 基于键值存储的rdf图结构存储和索引方法 | |
CN108319634B (zh) | 分布式文件***的目录访问方法和装置 | |
EP4064070A1 (en) | Cache optimization for data preparation | |
CN110275681A (zh) | 一种数据存储方法及数据存储*** | |
CN106201561A (zh) | 分布式缓存集群的升级方法与设备 | |
US20060236065A1 (en) | Method and system for variable dynamic memory management | |
US9239852B1 (en) | Item collections | |
CN109788013B (zh) | 分布式***中作业资源分配方法、装置及设备 | |
CN112035057B (zh) | 一种hive文件合并的方法和装置 | |
CN104750743A (zh) | 一种交易文件勾对***和方法 | |
CN108804571B (zh) | 一种数据存储方法、装置以及设备 | |
US7392359B2 (en) | Non-blocking distinct grouping of database entries with overflow | |
CN113590566B (zh) | 基于堆结构的SequenceFile存储优化方法、装置、设备及存储介质 | |
CN111752941A (zh) | 一种数据存储、访问方法、装置、服务器及存储介质 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
CB02 | Change of applicant information |
Address after: 430000 16-19 / F, building C3, future technology building, 999 Gaoxin Avenue, Donghu New Technology Development Zone, Wuhan, Hubei Province Applicant after: Wuhan dream database Co., Ltd Address before: 430000 16-19 / F, building C3, future technology building, 999 Gaoxin Avenue, Donghu New Technology Development Zone, Wuhan, Hubei Province Applicant before: WUHAN DAMENG DATABASE Co.,Ltd. |
|
CB02 | Change of applicant information | ||
CB03 | Change of inventor or designer information |
Inventor after: Mei Gang Inventor after: Gao Dongsheng Inventor after: Feng Yuan Inventor after: Hu Shuneng Inventor after: Chen Qi Inventor before: Fu Quan Inventor before: Mei Gang Inventor before: Gao Dongsheng |
|
CB03 | Change of inventor or designer information | ||
GR01 | Patent grant | ||
GR01 | Patent grant |