CN104331255A - 一种基于嵌入式文件***的流式数据读取方法 - Google Patents
一种基于嵌入式文件***的流式数据读取方法 Download PDFInfo
- Publication number
- CN104331255A CN104331255A CN201410653260.9A CN201410653260A CN104331255A CN 104331255 A CN104331255 A CN 104331255A CN 201410653260 A CN201410653260 A CN 201410653260A CN 104331255 A CN104331255 A CN 104331255A
- Authority
- CN
- China
- Prior art keywords
- task
- subtask
- data
- reading
- stream data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/176—Support for shared access to files; File sharing support
- G06F16/1767—Concurrency control, e.g. optimistic or pessimistic approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/148—File search processing
- G06F16/152—File search processing using file content signatures, e.g. hash values
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5066—Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Library & Information Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
Description
Claims (10)
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201410653260.9A CN104331255B (zh) | 2014-11-17 | 2014-11-17 | 一种基于嵌入式文件***的流式数据读取方法 |
PCT/CN2015/074082 WO2016078259A1 (zh) | 2014-11-17 | 2015-03-12 | 一种基于嵌入式文件***的流式数据读取方法 |
US15/527,323 US20170322948A1 (en) | 2014-11-17 | 2015-03-12 | Streaming data reading method based on embedded file system |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201410653260.9A CN104331255B (zh) | 2014-11-17 | 2014-11-17 | 一种基于嵌入式文件***的流式数据读取方法 |
Publications (2)
Publication Number | Publication Date |
---|---|
CN104331255A true CN104331255A (zh) | 2015-02-04 |
CN104331255B CN104331255B (zh) | 2018-04-17 |
Family
ID=52405990
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201410653260.9A Active CN104331255B (zh) | 2014-11-17 | 2014-11-17 | 一种基于嵌入式文件***的流式数据读取方法 |
Country Status (3)
Country | Link |
---|---|
US (1) | US20170322948A1 (zh) |
CN (1) | CN104331255B (zh) |
WO (1) | WO2016078259A1 (zh) |
Cited By (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2016078259A1 (zh) * | 2014-11-17 | 2016-05-26 | 中国科学院声学研究所 | 一种基于嵌入式文件***的流式数据读取方法 |
CN106598735A (zh) * | 2016-12-13 | 2017-04-26 | 广东金赋科技股份有限公司 | 一种分布式计算方法、主控节点、计算节点和*** |
WO2017092350A1 (zh) * | 2015-12-01 | 2017-06-08 | 乐视控股(北京)有限公司 | 提高缓存命中率的方法及装置 |
CN107547294A (zh) * | 2016-06-24 | 2018-01-05 | 财团法人电信技术中心 | 网络效能的测试***与测试方法 |
WO2018054200A1 (zh) * | 2016-09-26 | 2018-03-29 | 上海泓智信息科技有限公司 | 文件读取方法和装置 |
CN110516738A (zh) * | 2019-08-23 | 2019-11-29 | 佳都新太科技股份有限公司 | 一种分布式比对聚类方法、装置、电子设备及存储介质 |
CN110781159A (zh) * | 2019-10-28 | 2020-02-11 | 柏科数据技术(深圳)股份有限公司 | Ceph目录文件信息读取方法、装置、服务器及存储介质 |
CN110781137A (zh) * | 2019-10-28 | 2020-02-11 | 柏科数据技术(深圳)股份有限公司 | 分布式***的目录读取方法、装置、服务器和存储介质 |
CN113127443A (zh) * | 2020-01-14 | 2021-07-16 | 北京京东振世信息技术有限公司 | 一种更新缓存数据的方法和装置 |
CN113487026A (zh) * | 2021-07-05 | 2021-10-08 | 江苏号百信息服务有限公司 | 一种图计算中io节点高效读取数据的方法及*** |
WO2023077451A1 (zh) * | 2021-11-05 | 2023-05-11 | 中国科学院计算技术研究所 | 一种基于列存数据库的流式数据处理方法及*** |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10387207B2 (en) * | 2016-12-06 | 2019-08-20 | International Business Machines Corporation | Data processing |
CN111611105A (zh) * | 2020-05-15 | 2020-09-01 | 杭州涂鸦信息技术有限公司 | 一种对并发业务请求异步处理的优化方法及相关设备 |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5109511A (en) * | 1988-05-31 | 1992-04-28 | Hitachi, Ltd. | Shared resource managing method and system |
CN101656751A (zh) * | 2008-08-18 | 2010-02-24 | 北京数码大方科技有限公司 | 加速上传与下载文件的方法及其*** |
US20110145037A1 (en) * | 2009-12-16 | 2011-06-16 | Vertafore, Inc. | Document management method and apparatus to process a workflow task by parallel or serially processing subtasks thereof |
CN102467415A (zh) * | 2010-11-03 | 2012-05-23 | 大唐移动通信设备有限公司 | 一种业务面任务处理方法及设备 |
CN103942098A (zh) * | 2014-04-29 | 2014-07-23 | 国家电网公司 | 一种任务处理***和方法 |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101650669A (zh) * | 2008-08-14 | 2010-02-17 | 英业达股份有限公司 | 多线程下执行磁盘读写的方法 |
CN102368779B (zh) * | 2011-01-25 | 2013-04-17 | 麦克奥迪实业集团有限公司 | 一种用于移动互联网设备的超大图像加载显示方法 |
CN104331255B (zh) * | 2014-11-17 | 2018-04-17 | 中国科学院声学研究所 | 一种基于嵌入式文件***的流式数据读取方法 |
-
2014
- 2014-11-17 CN CN201410653260.9A patent/CN104331255B/zh active Active
-
2015
- 2015-03-12 US US15/527,323 patent/US20170322948A1/en not_active Abandoned
- 2015-03-12 WO PCT/CN2015/074082 patent/WO2016078259A1/zh active Application Filing
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5109511A (en) * | 1988-05-31 | 1992-04-28 | Hitachi, Ltd. | Shared resource managing method and system |
CN101656751A (zh) * | 2008-08-18 | 2010-02-24 | 北京数码大方科技有限公司 | 加速上传与下载文件的方法及其*** |
US20110145037A1 (en) * | 2009-12-16 | 2011-06-16 | Vertafore, Inc. | Document management method and apparatus to process a workflow task by parallel or serially processing subtasks thereof |
CN102467415A (zh) * | 2010-11-03 | 2012-05-23 | 大唐移动通信设备有限公司 | 一种业务面任务处理方法及设备 |
CN103942098A (zh) * | 2014-04-29 | 2014-07-23 | 国家电网公司 | 一种任务处理***和方法 |
Cited By (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2016078259A1 (zh) * | 2014-11-17 | 2016-05-26 | 中国科学院声学研究所 | 一种基于嵌入式文件***的流式数据读取方法 |
WO2017092350A1 (zh) * | 2015-12-01 | 2017-06-08 | 乐视控股(北京)有限公司 | 提高缓存命中率的方法及装置 |
CN107547294A (zh) * | 2016-06-24 | 2018-01-05 | 财团法人电信技术中心 | 网络效能的测试***与测试方法 |
WO2018054200A1 (zh) * | 2016-09-26 | 2018-03-29 | 上海泓智信息科技有限公司 | 文件读取方法和装置 |
CN106598735A (zh) * | 2016-12-13 | 2017-04-26 | 广东金赋科技股份有限公司 | 一种分布式计算方法、主控节点、计算节点和*** |
CN110516738B (zh) * | 2019-08-23 | 2022-09-16 | 佳都科技集团股份有限公司 | 一种分布式比对聚类方法、装置、电子设备及存储介质 |
CN110516738A (zh) * | 2019-08-23 | 2019-11-29 | 佳都新太科技股份有限公司 | 一种分布式比对聚类方法、装置、电子设备及存储介质 |
CN110781159A (zh) * | 2019-10-28 | 2020-02-11 | 柏科数据技术(深圳)股份有限公司 | Ceph目录文件信息读取方法、装置、服务器及存储介质 |
CN110781137A (zh) * | 2019-10-28 | 2020-02-11 | 柏科数据技术(深圳)股份有限公司 | 分布式***的目录读取方法、装置、服务器和存储介质 |
CN113127443A (zh) * | 2020-01-14 | 2021-07-16 | 北京京东振世信息技术有限公司 | 一种更新缓存数据的方法和装置 |
CN113487026A (zh) * | 2021-07-05 | 2021-10-08 | 江苏号百信息服务有限公司 | 一种图计算中io节点高效读取数据的方法及*** |
CN113487026B (zh) * | 2021-07-05 | 2024-05-03 | 江苏号百科技有限公司 | 一种图计算中io节点高效读取数据的方法及*** |
WO2023077451A1 (zh) * | 2021-11-05 | 2023-05-11 | 中国科学院计算技术研究所 | 一种基于列存数据库的流式数据处理方法及*** |
Also Published As
Publication number | Publication date |
---|---|
WO2016078259A1 (zh) | 2016-05-26 |
US20170322948A1 (en) | 2017-11-09 |
CN104331255B (zh) | 2018-04-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN104331255A (zh) | 一种基于嵌入式文件***的流式数据读取方法 | |
JP7271059B2 (ja) | マルチクラスタウェアハウス | |
US9344750B2 (en) | Method and system for scheduling commercial advertisement | |
US9760486B2 (en) | Accelerating cache state transfer on a directory-based multicore architecture | |
CN102523279B (zh) | 一种分布式文件***及其热点文件存取方法 | |
JP2007518174A5 (zh) | ||
CN107729020A (zh) | 一种实现大规模容器快速部署的方法 | |
CN106294193B (zh) | 存储设备及基于该存储设备的分块存储方法 | |
CN102710763B (zh) | 一种分布式缓存池化、分片及故障转移的方法及*** | |
CN102084331A (zh) | 在多处理器/多线程环境下协调存储请求的装置、***和方法 | |
JP2012511773A (ja) | 効率的なデータ同期化のための方法及びシステム | |
KR20110074489A (ko) | 데이터 즉시 청킹을 사용하여 파일 입출력을 스케줄 하는 방법 | |
CN109164981B (zh) | 磁盘管理方法、装置、存储介质和设备 | |
CN102591970A (zh) | 一种分布式键-值查询方法和查询引擎*** | |
US11271800B1 (en) | Leaderless, parallel, and topology-aware protocol for achieving consensus with recovery from failure of all nodes in a group | |
CN102955817A (zh) | 一种基于共享内存的对象管理与恢复方法及*** | |
CN103593243A (zh) | 一种动态可扩展的增加虚拟机资源的方法 | |
CN113010496B (zh) | 一种数据迁移方法、装置、设备和存储介质 | |
CN106959893A (zh) | 加速器、用于加速器的内存管理方法以及数据处理*** | |
CN105389134A (zh) | 一种闪存接口控制方法及装置 | |
US20110078383A1 (en) | Cache Management for Increasing Performance of High-Availability Multi-Core Systems | |
CN100372283C (zh) | 多点下载中保证数据块在整个***均匀分布的方法和装置 | |
WO2018233332A1 (zh) | 一种分布式存储内存管理方法、***及计算机存储介质 | |
CN104202371B (zh) | 一种基于iSCSI块直存的跨设备扩容方法和装置 | |
JP5742614B2 (ja) | リストア処理プログラム、リストア方法、及び情報処理装置 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant | ||
TR01 | Transfer of patent right |
Effective date of registration: 20210813 Address after: Room 1601, 16th floor, East Tower, Ximei building, No. 6, Changchun Road, high tech Industrial Development Zone, Zhengzhou, Henan 450001 Patentee after: Zhengzhou xinrand Network Technology Co.,Ltd. Address before: 100190, No. 21 West Fourth Ring Road, Beijing, Haidian District Patentee before: INSTITUTE OF ACOUSTICS, CHINESE ACADEMY OF SCIENCES Effective date of registration: 20210813 Address after: 100190, No. 21 West Fourth Ring Road, Beijing, Haidian District Patentee after: INSTITUTE OF ACOUSTICS, CHINESE ACADEMY OF SCIENCES Address before: 100190, No. 21 West Fourth Ring Road, Beijing, Haidian District Patentee before: INSTITUTE OF ACOUSTICS, CHINESE ACADEMY OF SCIENCES Patentee before: BEIJING INTELLIX TECHNOLOGIES Co.,Ltd. |
|
TR01 | Transfer of patent right |