CN110263059A - Spark-Streaming中间数据分区方法、装置、计算机设备和存储介质 - Google Patents
Spark-Streaming中间数据分区方法、装置、计算机设备和存储介质 Download PDFInfo
- Publication number
- CN110263059A CN110263059A CN201910438036.0A CN201910438036A CN110263059A CN 110263059 A CN110263059 A CN 110263059A CN 201910438036 A CN201910438036 A CN 201910438036A CN 110263059 A CN110263059 A CN 110263059A
- Authority
- CN
- China
- Prior art keywords
- subregion
- cluster
- frequency weight
- updated
- intermediate data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24554—Unary operations; Data partitioning operations
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Complex Calculations (AREA)
Abstract
Description
Claims (10)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910438036.0A CN110263059B (zh) | 2019-05-24 | 2019-05-24 | Spark-Streaming中间数据分区方法、装置、计算机设备和存储介质 |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910438036.0A CN110263059B (zh) | 2019-05-24 | 2019-05-24 | Spark-Streaming中间数据分区方法、装置、计算机设备和存储介质 |
Publications (2)
Publication Number | Publication Date |
---|---|
CN110263059A true CN110263059A (zh) | 2019-09-20 |
CN110263059B CN110263059B (zh) | 2021-05-11 |
Family
ID=67915335
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201910438036.0A Active CN110263059B (zh) | 2019-05-24 | 2019-05-24 | Spark-Streaming中间数据分区方法、装置、计算机设备和存储介质 |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN110263059B (zh) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111258624A (zh) * | 2020-01-13 | 2020-06-09 | 上海交通大学 | 开源软件开发中Issue解决时间的预测方法及*** |
CN112000467A (zh) * | 2020-07-24 | 2020-11-27 | 广东技术师范大学 | 一种数据倾斜处理方法、装置、终端设备及存储介质 |
CN112612614A (zh) * | 2020-12-28 | 2021-04-06 | 江苏苏宁云计算有限公司 | 一种数据排序方法、装置及*** |
CN113626426A (zh) * | 2021-07-06 | 2021-11-09 | 佛山市禅城区政务服务数据管理局 | 一种生态网格数据的采集传输方法及*** |
CN112612614B (zh) * | 2020-12-28 | 2024-07-26 | 江苏苏宁云计算有限公司 | 一种数据排序方法、装置及*** |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130238621A1 (en) * | 2012-03-06 | 2013-09-12 | Microsoft Corporation | Entity Augmentation Service from Latent Relational Data |
CN105550374A (zh) * | 2016-01-29 | 2016-05-04 | 湖南大学 | Spark云服务环境下面向大数据的随机森林并行机器学习方法 |
US20160321350A1 (en) * | 2013-12-27 | 2016-11-03 | International Business Machines Corporation | Stratified sampling using adaptive parallel data processing |
CN109034981A (zh) * | 2018-08-23 | 2018-12-18 | 上海海事大学 | 一种电商协同过滤推荐方法 |
-
2019
- 2019-05-24 CN CN201910438036.0A patent/CN110263059B/zh active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130238621A1 (en) * | 2012-03-06 | 2013-09-12 | Microsoft Corporation | Entity Augmentation Service from Latent Relational Data |
US20160321350A1 (en) * | 2013-12-27 | 2016-11-03 | International Business Machines Corporation | Stratified sampling using adaptive parallel data processing |
CN105550374A (zh) * | 2016-01-29 | 2016-05-04 | 湖南大学 | Spark云服务环境下面向大数据的随机森林并行机器学习方法 |
CN109034981A (zh) * | 2018-08-23 | 2018-12-18 | 上海海事大学 | 一种电商协同过滤推荐方法 |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111258624A (zh) * | 2020-01-13 | 2020-06-09 | 上海交通大学 | 开源软件开发中Issue解决时间的预测方法及*** |
CN111258624B (zh) * | 2020-01-13 | 2023-04-28 | 上海交通大学 | 开源软件开发中Issue解决时间的预测方法及*** |
CN112000467A (zh) * | 2020-07-24 | 2020-11-27 | 广东技术师范大学 | 一种数据倾斜处理方法、装置、终端设备及存储介质 |
CN112612614A (zh) * | 2020-12-28 | 2021-04-06 | 江苏苏宁云计算有限公司 | 一种数据排序方法、装置及*** |
CN112612614B (zh) * | 2020-12-28 | 2024-07-26 | 江苏苏宁云计算有限公司 | 一种数据排序方法、装置及*** |
CN113626426A (zh) * | 2021-07-06 | 2021-11-09 | 佛山市禅城区政务服务数据管理局 | 一种生态网格数据的采集传输方法及*** |
Also Published As
Publication number | Publication date |
---|---|
CN110263059B (zh) | 2021-05-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN115220918A (zh) | 一种神经网络的内存分配方法及装置 | |
Shahvari et al. | Hybrid flow shop batching and scheduling with a bi-criteria objective | |
CN110263059A (zh) | Spark-Streaming中间数据分区方法、装置、计算机设备和存储介质 | |
Kabiljo et al. | Social hash partitioner: a scalable distributed hypergraph partitioner | |
Rahimi-Vahed et al. | Fleet-sizing for multi-depot and periodic vehicle routing problems using a modular heuristic algorithm | |
CN110602227B (zh) | 一种智能合约管理的方法以及相关装置 | |
CN112101674B (zh) | 基于群智能算法资源配置匹配方法、装置、设备及介质 | |
Kaya et al. | Iterative-improvement-based heuristics for adaptive scheduling of tasks sharing files on heterogeneous master-slave environments | |
CN110941602A (zh) | 数据库的配置方法、装置、电子设备及存储介质 | |
CN108170602A (zh) | 一种测试用例生成方法、装置、终端及计算机可读介质 | |
CN112445702A (zh) | 基于蚁群算法的自动化测试方法和*** | |
CN110969354A (zh) | 线性流程配置方法、装置、计算机设备及存储介质 | |
CN113138849B (zh) | 一种计算资源调度和迁移方法、相关装置及*** | |
CN104778088A (zh) | 一种基于减少进程间通信开销的并行i/o优化方法与*** | |
CN106202374A (zh) | 一种数据处理方法及装置 | |
CN108345603A (zh) | 一种sql语句解析方法及装置 | |
CN115361340A (zh) | Ab实验分流方法、装置、计算机设备和存储介质 | |
Khan et al. | Fast graph partitioning algorithms | |
Seiferth et al. | Offsite autotuning approach: performance model driven autotuning applied to parallel explicit ODE methods | |
US20080147221A1 (en) | Grid modeling tool | |
Li et al. | A sort-based interest matching algorithm with two exclusive judging conditions for region overlap | |
Vescan et al. | A hybrid evolutionary multiobjective approach for the dynamic component selection problem | |
CN109739638A (zh) | 一种基于深度学习的edf可调度性判定方法及装置 | |
Duin | Two fast algorithms for all-pairs shortest paths | |
CN114265556B (zh) | 一种数据存储方法及装置 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant | ||
CB03 | Change of inventor or designer information | ||
CB03 | Change of inventor or designer information |
Inventor after: Tang Zhuo Inventor after: Fu Zhongming Inventor after: Chen Cen Inventor after: Chen Jianguo Inventor after: Li Kenli Inventor after: Li Keqin Inventor before: Tang Zhuo Inventor before: Fu Zhongming Inventor before: Chen Cen Inventor before: Chen Jianguo Inventor before: Li Kenli Inventor before: Li Keqin Inventor before: Liao Xiangke |
|
CB03 | Change of inventor or designer information | ||
CB03 | Change of inventor or designer information |
Inventor after: Li Kenli Inventor after: Fu Zhongming Inventor after: Tang Zhuo Inventor after: Chen Cen Inventor after: Chen Jianguo Inventor after: Li Keqin Inventor before: Tang Zhuo Inventor before: Fu Zhongming Inventor before: Chen Cen Inventor before: Chen Jianguo Inventor before: Li Kenli Inventor before: Li Keqin |