WO2015154549A1 - 数据的处理方法及装置 - Google Patents

数据的处理方法及装置 Download PDF

Info

Publication number
WO2015154549A1
WO2015154549A1 PCT/CN2015/070235 CN2015070235W WO2015154549A1 WO 2015154549 A1 WO2015154549 A1 WO 2015154549A1 CN 2015070235 W CN2015070235 W CN 2015070235W WO 2015154549 A1 WO2015154549 A1 WO 2015154549A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
processed
top box
distributed
kpi
Prior art date
Application number
PCT/CN2015/070235
Other languages
English (en)
French (fr)
Inventor
丁岩
陈斌
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Publication of WO2015154549A1 publication Critical patent/WO2015154549A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor

Definitions

  • the present invention relates to the field of communications, and in particular to a method and apparatus for processing data.
  • OTT TV is an abbreviation of “Over The Top TV”, which refers to an open Internet-based video service.
  • the terminal can be a TV, a computer, a set-top box, a PAD, a smart phone, etc., which provides services on the network, emphasizing service and physics. Network independence.
  • OTT TV is an Internet TV that meets the needs of consumers and integrates full-featured Internet TV with interactive TV functions.
  • OTT TV refers to the convergence of Internet Protocol (IP) video and Internet applications for television transmission over the public Internet.
  • IP Internet Protocol
  • the receiving terminal is an Internet TV integrated machine or a set top box + a television set.
  • OTT TV refers to the controllable and manageable service of video content provided by state-owned broadcasters through the public Internet for television.
  • the receiving terminal is generally a domestic Internet TV integrated machine.
  • OTT TVs in the related art mainly have two implementation modes: HTTP Progressive Download (HPD) and HTTP Adaptive Streaming (HAS).
  • HPD HTTP Progressive Download
  • HAS HTTP Adaptive Streaming
  • Traditional OTT TVs generally use HPD technology.
  • the HPD-based client only needs to wait a short period of time to download and buffer the first part of the media file before starting playback, and then play it while downloading.
  • HPD OTT TV has many limitations, such as: it is not suitable for the transmission of live programs with high real-time requirements; the waiting delay for initial playback is generally longer; when the network bandwidth is unstable, the card screen phenomenon is more likely to occur;
  • the video file will continue to be downloaded, and when the user gives up the program to watch, it will cause a waste of downloaded files (consuming bandwidth).
  • HAS-based OTT TV uses video segmentation and adaptive bit rate (ABR) technology.
  • ABR adaptive bit rate
  • the media stream splitter splits the video stream output by the encoder into a series of consecutive, equal-length small slice files and stores them in a web content distribution server.
  • the HAS client device can automatically request the appropriate video quality (ie different resolution and bit rate) fragment files from the web server based on the available bandwidth, thus giving the user the best visual experience.
  • HAS video In order to facilitate the HAS client to achieve fast, real-time switching between different code rate slices, HAS video generally uses a shorter slice length (for example: 10 seconds). Because the HAS system can be The same size screen terminal provides video fragmentation files suitable for resolution and smooth video playback under different network bandwidth conditions. Therefore, HAS is considered by the industry to be the core technology of multi-screen interactive video that is ubiquitous in the future.
  • the embodiment of the invention provides a method and a device for processing data, so as to at least solve the problem of how to monitor and analyze the data reported by the OTT set-top box by using the big data technology in the related art.
  • a method of processing data is provided.
  • the data processing method includes: the data to be processed reported by the receiver top box, wherein the data to be processed is obtained by processing, by the set top box, the original data obtained from the third party application component, and the original data includes the following. At least one of: file download data, user triggered behavior data, file play data; analysis of processed data to obtain key performance indicator (KPI) data.
  • KPI key performance indicator
  • the data to be processed is analyzed, and obtaining the KPI data comprises: parsing the data to be processed by using a load balancing component, and sending the parsed data to the distributed message queue cluster; and utilizing the distributed message queue in the distributed message queue cluster
  • the parsed data is backed up, and the parsed data is sent to the distributed real-time computing system; the parsed data is analyzed and counted by the distributed real-time computing system to obtain KPI data.
  • the parsing of the data to be processed by the load balancing component comprises: parsing the message header and the message body from the data to be processed by using the load balancing component; and stripping the message header by using the load balancing component to reserve the message body.
  • the data to be processed reported by the top box is received via the negotiated data interface according to the preset duration.
  • the method further comprises: performing offline analysis mining on the KPI data.
  • a data processing apparatus is provided.
  • the data processing apparatus includes: a receiving module configured to report data to be processed reported by the receiver set top box, wherein the data to be processed is input by the set top box to the original data acquired from the third party application component.
  • the original data includes at least one of the following: file download data, user trigger behavior data, file play data; processing module, set to analyze the data to be processed, and obtain KPI data.
  • the processing module comprises: a parsing unit configured to parse the data to be processed by the load balancing component, and send the parsed data to the distributed message queue cluster; and the backup unit is configured to utilize the distribution in the distributed message queue cluster The message queue backs up the parsed data, and sends the parsed data to the distributed real-time computing system; the analysis unit is configured to analyze and collect the parsed data through the distributed real-time computing system to obtain KPI data.
  • the parsing unit comprises: a parsing subunit, configured to parse the message header and the message body from the to-be-processed data by using the load balancing component; and the processing subunit is configured to perform stripping processing on the message header by using the load balancing component, and retain the message body .
  • the receiving module is configured to process the data to be processed reported by the top box via the negotiated data interface according to the preset duration.
  • the above apparatus further comprises: an analysis module, configured to perform offline analysis and mining on the KPI data.
  • the to-be-processed data reported by the receiver set-top box is used, wherein the to-be-processed data is obtained by processing, by the set-top box, the original data obtained from the third-party application component, and the original data includes at least one of the following: Download data, user triggered behavior data, file playback data; analyze the data to be processed, obtain KPI data, and solve the problem of how to use the big data technology to monitor and analyze the data reported by the OTT set-top box in related technologies, and then it can be timely and timely
  • the massive raw data reported by the OTT set-top box is statistically analyzed, and offline mining can be realized.
  • FIG. 1 is a flow chart of a method of processing data according to an embodiment of the present invention
  • FIG. 2 is a schematic diagram of an OTT set top box data collection model in accordance with a preferred embodiment of the present invention
  • FIG. 3 is a flow chart of data processing based on the OTT set top box data collection model shown in FIG. 2, in accordance with a preferred embodiment of the present invention
  • FIG. 4 is a block diagram showing the structure of a data processing apparatus according to an embodiment of the present invention.
  • FIG. 5 is a block diagram showing the structure of a data processing apparatus in accordance with a preferred embodiment of the present invention.
  • FIG. 1 is a flow chart of a method of processing data according to an embodiment of the present invention. As shown in FIG. 1, the method may include the following processing steps:
  • Step S102 The data to be processed reported by the receiver top box, wherein the data to be processed is obtained by processing, by the set top box, the original data obtained from the third party application component, and the original data includes at least one of the following: file download data, user Trigger behavior data, file playback data;
  • Step S104 analyzing the data to be processed, and acquiring KPI data.
  • big data technology cannot be used to monitor and analyze the data reported by the OTT set-top box.
  • the method shown in FIG. 1 is used to process the raw data obtained from the third-party application component by the set-top box, and then the data to be processed is obtained, and then the data to be processed reported by the set-top box is analyzed to obtain KPI data, thereby solving the problem.
  • the data to be processed reported by the top box can be received via the negotiated data interface according to the preset duration.
  • the third party application component collects file downloads, user triggered behavior, and plays back related raw data, and transmits the collected raw data to the set top box data collection interface in real time.
  • the set-top box processes the collected raw data, and then reports the data to the specified address via the negotiated data interface according to the preset duration.
  • step S104 the data to be processed is analyzed, and acquiring the KPI data may include the following operations:
  • Step S1 parsing the data to be processed by using the load balancing component, and sending the parsed data to the distributed message queue cluster;
  • Step S2 Backing up the parsed data by using a distributed message queue in the distributed message queue cluster, and sending the parsed data to the distributed real-time computing system;
  • Step S3 Perform analysis and statistics on the parsed data through a distributed real-time computing system to obtain KPI data.
  • a load balancing component can be employed to receive data reported by the set-top box.
  • the data reported by the set top box is parsed, and at the same time, as a client of the distributed message queue, the message body is sent to the distributed message queue cluster.
  • the distributed message queue backs up the received data and sends it to the distributed real-time computing system.
  • the distributed real-time computing system uses streaming technology to perform real-time parsing and statistics on valid data, thereby obtaining statistical KPI data of the entire network.
  • parsing the data to be processed by using the load balancing component may include the following steps:
  • Step S11 using a load balancing component to parse the message header and the message body from the to-be-processed data;
  • Step S12 The message header is stripped by the load balancing component, and the message body is reserved.
  • the load balancing component can be used to parse the data reported by the set top box, stripping the message headers, leaving only the actual useful message body.
  • step S104 the data to be processed is analyzed, and after the KPI data is acquired, the following operations may also be included:
  • Step S4 Perform offline analysis and mining on the KPI data.
  • the distributed real-time computing system can use the streaming technology to perform real-time parsing and statistics on the valid data, thereby obtaining the statistical KPI data of the entire network, and then permanently storing the KPI data to the distributed file system.
  • the off-line analysis system can perform on-line analysis and mining of the massive data collected above at any time as needed.
  • FIG. 2 is a schematic diagram of an OTT set top box data collection model in accordance with a preferred embodiment of the present invention.
  • the set-top box is preset with a third-party application and a platform-side probe component, and the preset monitoring center reports the address.
  • the third-party application component is mainly responsible for interacting with the player and the electronic program menu (EPG), collecting user trigger events, the most original playback data, and the like.
  • EPG electronic program menu
  • licensees have application components that have these functions.
  • the probe component is part of the monitoring and analysis system, and the raw data collected by the licensee can be analyzed and processed, and can be upgraded separately.
  • the technical solution provided by the embodiment of the present invention will be applied in a monitoring center.
  • the operator can decide whether to remotely modify the reporting address of the monitoring center through the network management. Due to various reasons such as network planning, in some cases, operators need to change the allocation to monitoring. The relevant IP address of the center. At this point, you must modify the monitoring center address of the set-top box in time, otherwise the monitoring center may not collect the original data.
  • the load balancing component of the monitoring center performs the detection of legitimate terminals and the parsing of network data message headers.
  • the data reported by the illegal terminal will be discarded, and the basic information of the terminal will be recorded, which is convenient for troubleshooting.
  • the distributed real-time computing system initiates separate threads for detailed data storage without affecting real-time monitoring and analysis.
  • FIG. 3 is a flow diagram of data processing based on the OTT set top box data collection model shown in FIG. 2, in accordance with a preferred embodiment of the present invention. As shown in FIG. 3, the method may include the following steps:
  • the third-party application component collects the file download, the user triggers the behavior, and plays the relevant original data, and transmits the collected original data to the set-top box data collection interface in real time, for example, if the user has clicked the pause or cancel the pause button, Record the time when the user clicks to pause or cancel the pause button; if buffering has already occurred during the playback of the video, the buffer start time and buffer time can be recorded.
  • the set top box performs processing calculation on the collected original data, and reports the data to the specified address via the negotiated data interface according to a preset duration (1 minute, 30 seconds or even 10 seconds).
  • the load balancing component is used to receive data reported by the top box. Parsing the data reported by the set-top box, stripping the message header, leaving only the actual useful message body, and sending the message body to the distributed message queue cluster as the client of the distributed message queue.
  • the distributed message queue backs up the received data (preventing loss) and simultaneously sends it to the distributed real-time computing system that has registered to listen.
  • the fifth step uses the streaming technology to analyze and count the effective data in real time, and basically obtains the statistical key performance indicator (KPI) data of the whole network in real time, and restores the viewing behavior of the user.
  • KPI statistical key performance indicator
  • massive amounts of raw data are also stored faster in distributed file systems.
  • the sixth step is to perform on-line analysis and mining of the massive data collected above according to the needs. For example, the source of the film is classified into a finer classification, and the viewing behavior of each user is analyzed, and the interests and hobbies of the user are found, so that the newly released similar film source is pushed to the user; the program that the user last viewed and the viewing are recorded. Where to go, you can continue playing from that location the next time you need to continue watching.
  • the processing device of the data may include: a receiving module 10 configured to process data to be processed reported by the receiver set-top box, wherein the data to be processed is processed by the set-top box for the original data acquired from the third-party application component.
  • the start data includes at least one of the following: file download data, user trigger behavior data, file play data; the processing module 20 is configured to analyze the data to be processed, and obtain KPI data.
  • the device shown in FIG. 4 is used to solve the problem of how to use the big data technology to monitor and analyze the data reported by the OTT set-top box in the related art, and then the statistical analysis of the massive raw data reported by the OTT set-top box can be performed in a timely manner, and Implement offline mining.
  • the processing module 20 may include: a parsing unit 200 configured to parse the data to be processed by the load balancing component, and send the parsed data to the distributed message queue cluster; the backup unit 202, set The data is backed up by using the distributed message queue in the distributed message queue cluster, and the parsed data is sent to the distributed real-time computing system; the analyzing unit 204 is configured to parse through the distributed real-time computing system. The data is analyzed and statistically obtained to obtain KPI data.
  • a parsing unit 200 configured to parse the data to be processed by the load balancing component, and send the parsed data to the distributed message queue cluster
  • the backup unit 202 set The data is backed up by using the distributed message queue in the distributed message queue cluster, and the parsed data is sent to the distributed real-time computing system
  • the analyzing unit 204 is configured to parse through the distributed real-time computing system. The data is analyzed and statistically obtained to obtain KPI data.
  • the parsing unit 200 may include: a parsing subunit (not shown) configured to parse the message header and the message body from the to-be-processed data by using a load balancing component; Not shown), set to use the load balancing component to strip the message header and retain the message body.
  • a parsing subunit (not shown) configured to parse the message header and the message body from the to-be-processed data by using a load balancing component; Not shown), set to use the load balancing component to strip the message header and retain the message body.
  • the receiving module 10 is configured to process the data to be processed reported by the top box via the negotiated data interface according to the preset duration.
  • the foregoing apparatus further includes: an analysis module 30 configured to perform offline analysis mining on the KPI data.
  • the foregoing embodiment achieves the following technical effects (it is required that the effects are achievable by some preferred embodiments): the technical solution provided by the embodiment of the present invention is adopted by the operator, The Internet TV licensee and the set-top box manufacturer cooperated to realize the collection and quasi-real-time reporting of the user's viewing behavior data and effect data, and then monitor and analyze the massive data reported by the big data technology, thereby restoring the user's viewing behavior and discovering.
  • the problems existing in the viewing and the analysis of the user's viewing interests and hobbies so that relevant parties can improve in time, improve the quality of service and conduct targeted marketing, and then statistically analyze the massive raw data reported by the OTT set-top box in a timely manner, and Offline mining is possible.
  • the terminal data collection process is jointly participated by operators, domestic mainstream licensees, and mainstream set-top box manufacturers, it can be promoted as an industry standard.
  • modules or steps of the present invention described above can be implemented by a general-purpose computing device that can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they may be implemented by program code executable by the computing device such that they may be stored in the storage device by the computing device and, in some cases, may be different from The steps shown or described are performed sequentially, or they are separately fabricated into individual integrated circuit modules, or a plurality of modules or steps thereof are fabricated into a single integrated circuit module. Thus, the invention is not limited to any specific combination of hardware and software.
  • the data processing method and apparatus have the following beneficial effects: through the cooperation of the operator, the Internet TV license party, and the set-top box manufacturer, the user's viewing behavior data and the effect data collection and quasi-realization are realized.
  • Real-time reporting and then through the big data technology to monitor and analyze the massive data reported, which can restore the user's viewing behavior, discover the problems in the viewing and analyze the user's viewing interests, so that relevant parties can improve and improve the service in time.
  • the quality is targeted and marketed, so that the massive raw data reported by the OTT set-top box can be statistically analyzed in a timely manner, and offline mining can be realized.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

本发明公开了一种数据的处理方法及装置，在上述方法中，接收机顶盒上报的待处理数据，其中，待处理数据是由机顶盒对从第三方应用组件获取到的原始数据进行加工计算后得到的，原始数据包括以下至少之一：文件下载数据、用户触发行为数据、文件播放数据；对待处理数据进行分析，获取KPI数据。根据本发明提供的技术方案，可以较为及时地对OTT机顶盒上报的海量原始数据进行统计分析。

Description

数据的处理方法及装置 技术领域
本发明涉及通信领域，具体而言，涉及一种数据的处理方法及装置。
背景技术
随着互联网技术的飞速发展，互联网电视即OTT TV发展迅速，智能机顶盒越来越普及。OTT TV是“Over The Top TV”的缩写，是指基于开放互联网的视频服务，终端可以是电视机、电脑、机顶盒、PAD、智能手机等等，其在网络之上提供服务，强调服务与物理网络的无关性。从消费者的角度出发，OTT TV就是互联网电视，满足消费者的需求，集成互动电视功能的全功能的互联网电视。
在国际上，OTT TV指通过公共互联网面向电视传输的互联网协议(IP)视频和互联网应用融合的服务。其接收终端为互联网电视一体机或机顶盒+电视机。而在我国，OTT TV是指通过公共互联网面向电视机传输的由国有广播电视机构提供视频内容的可控可管服务。接收终端一般为国产互联网电视一体机。
目前，相关技术中的OTT TV主要有两种实现方式：HTTP渐进下载(HTTP Progressive Download，简称为HPD)和HTTP自适应流媒体(HTTP Adaptive Streaming，简称为HAS)。传统的OTT TV一般采用HPD技术。基于HPD的客户端在开始播放之前仅需等待一段较短的时间用于下载和缓冲媒体文件最前面的一部分数据，之后便可以一边下载一边播放。HPD OTT TV存在诸多的局限性，例如：不适合对实时性要求较高的直播节目的传输；初始播放的等待时延一般较长；当网络带宽不稳定时比较容易出现卡屏现象；由于客户端会持续下载视频文件，当用户中途放弃节目观看，会造成已下载文件(消耗带宽)的浪费。
为了克服HPD OTT TV技术的局限性，近年来基于HAS的OTT TV技术逐渐被业界广泛采用和推广。HAS OTT TV采用视频分片和自适应码率(ABR)技术。在HAS***中，媒体流分割器将编码器输出的视频流分割为一系列连续的、长度均等的小分片文件，并将它们存储在Web内容分发服务器。HAS客户端设备能够在可用的带宽的基础上，自动向Web服务器请求合适的视频质量(即不同的分辨率和码率)的分片文件，从而给用户最好的视觉体验。为了便于HAS客户端实现不同码率分片之间的快速、实时切换，HAS视频一般采用较短的分片长度(例如：10秒)。由于HAS***可向不 同屏幕大小的终端提供适合分辨率的视频分片文件，并可在不同网络带宽情况下实现流畅的视频播放，因此HAS被业内认为是未来无所不在的多屏互动视频的核心技术。
然而，尽管OTT TV技术的不断发展带来诸多益处，但是，如何能够有效地监控OTT用户的播放质量是运营商无法回避的问题，目前相关技术中也缺乏有效的解决方案。
发明内容
本发明实施例提供了一种数据的处理方法及装置，以至少解决相关技术中如何利用大数据技术对OTT机顶盒上报的数据进行监控分析的问题。
根据本发明的一个方面，提供了一种数据的处理方法。
根据本发明实施例的数据的处理方法包括：接收机顶盒上报的待处理数据，其中，待处理数据是由机顶盒对从第三方应用组件获取到的原始数据进行加工计算后得到的，原始数据包括以下至少之一：文件下载数据、用户触发行为数据、文件播放数据；对待处理数据进行分析，获取关键性能指标(KPI)数据。
优选地，对待处理数据进行分析，获取KPI数据包括：采用负载均衡组件对待处理数据进行解析，并将解析后的数据发送至分布式消息队列集群；利用分布式消息队列集群中的分布式消息队列对解析后的数据进行备份，并将解析后的数据发送至分布式实时计算***；通过分布式实时计算***对解析后的数据进行分析统计，得到KPI数据。
优选地，采用负载均衡组件对待处理数据进行解析包括：采用负载均衡组件从待处理数据中解析出消息头和消息体；采用负载均衡组件将消息头进行剥离处理，保留消息体。
优选地，按照预设时长经由协商的数据接口接收机顶盒上报的待处理数据。
优选地，在对待处理数据进行分析，获取KPI数据之后，还包括：对KPI数据进行离线分析挖掘。
根据本发明的另一方面，提供了一种数据的处理装置。
根据本发明实施例的数据的处理装置包括：接收模块，设置为接收机顶盒上报的待处理数据，其中，待处理数据是由机顶盒对从第三方应用组件获取到的原始数据进 行加工计算后得到的，原始数据包括以下至少之一：文件下载数据、用户触发行为数据、文件播放数据；处理模块，设置为对待处理数据进行分析，获取KPI数据。
优选地，处理模块包括：解析单元，设置为采用负载均衡组件对待处理数据进行解析，并将解析后的数据发送至分布式消息队列集群；备份单元，设置为利用分布式消息队列集群中的分布式消息队列对解析后的数据进行备份，并将解析后的数据发送至分布式实时计算***；分析单元，设置为通过分布式实时计算***对解析后的数据进行分析统计，得到KPI数据。
优选地，解析单元包括：解析子单元，设置为采用负载均衡组件从待处理数据中解析出消息头和消息体；处理子单元，设置为采用负载均衡组件将消息头进行剥离处理，保留消息体。
优选地，接收模块，设置为按照预设时长经由协商的数据接口接收机顶盒上报的待处理数据。
优选地，上述装置还包括：分析模块，设置为对KPI数据进行离线分析挖掘。
通过本发明实施例，采用接收机顶盒上报的待处理数据，其中，待处理数据是由机顶盒对从第三方应用组件获取到的原始数据进行加工计算后得到的，原始数据包括以下至少之一：文件下载数据、用户触发行为数据、文件播放数据；对待处理数据进行分析，获取KPI数据，解决了相关技术中如何利用大数据技术对OTT机顶盒上报的数据进行监控分析的问题，进而可以较为及时地对OTT机顶盒上报的海量原始数据进行统计分析，并且可以实现离线挖掘。
附图说明
此处所说明的附图用来提供对本发明的进一步理解，构成本申请的一部分，本发明的示意性实施例及其说明用于解释本发明，并不构成对本发明的不当限定。在附图中：
图1是根据本发明实施例的数据的处理方法的流程图；
图2是根据本发明优选实施例的OTT机顶盒数据采集模型的示意图；
图3是根据本发明优选实施例的基于图2所示的OTT机顶盒数据采集模型的数据处理的流程图；
图4是根据本发明实施例的数据的处理装置的结构框图；
图5是根据本发明优选实施例的数据的处理装置的结构框图。
具体实施方式
下文中将参考附图并结合实施例来详细说明本发明。需要说明的是，在不冲突的情况下，本申请中的实施例及实施例中的特征可以相互组合。
图1是根据本发明实施例的数据的处理方法的流程图。如图1所示，该方法可以包括以下处理步骤：
步骤S102：接收机顶盒上报的待处理数据，其中，待处理数据是由机顶盒对从第三方应用组件获取到的原始数据进行加工计算后得到的，原始数据包括以下至少之一：文件下载数据、用户触发行为数据、文件播放数据；
步骤S104：对待处理数据进行分析，获取KPI数据。
相关技术中无法利用大数据技术对OTT机顶盒上报的数据进行监控分析。采用如图1所示的方法，通过机顶盒在对从第三方应用组件获取到的原始数据进行加工计算后得到待处理数据，再对机顶盒上报的待处理数据进行分析进而获取KPI数据，由此解决了相关技术中如何利用大数据技术对OTT机顶盒上报的数据进行监控分析的问题，进而可以较为及时地对OTT机顶盒上报的海量原始数据进行统计分析，并且可以实现离线挖掘。
在优选实施过程中，可以按照预设时长经由协商的数据接口接收机顶盒上报的待处理数据。
在优选实施例中，第三方应用组件收集文件下载、用户触发行为以及播放有关的原始数据，并向机顶盒数据采集接口实时传输收集到的原始数据。机顶盒对采集到的原始数据进行加工计算，然后再按照预设时长经由协商的数据接口将数据上报至指定的地址。
优选地，在步骤S104中，对待处理数据进行分析，获取KPI数据可以包括以下操作：
步骤S1：采用负载均衡组件对待处理数据进行解析，并将解析后的数据发送至分布式消息队列集群；
步骤S2：利用分布式消息队列集群中的分布式消息队列对解析后的数据进行备份，并将解析后的数据发送至分布式实时计算***；
步骤S3：通过分布式实时计算***对解析后的数据进行分析统计，得到KPI数据。
在优选实施例中，可以采用负载均衡组件来接收机顶盒上报的数据。对机顶盒上报的数据进行解析，同时作为分布式消息队列的客户端，将消息体发送至分布式消息队列集群。分布式消息队列对接收到的数据进行备份，并发送至分布式实时计算***。分布式实时计算***利用流技术对有效数据进行实时解析与统计，从而得到全网的统计KPI数据。
优选地，在步骤S1中，采用负载均衡组件对待处理数据进行解析可以包括以下步骤：
步骤S11：采用负载均衡组件从待处理数据中解析出消息头和消息体；
步骤S12：采用负载均衡组件将消息头进行剥离处理，保留消息体。
在优选实施例中，可以采用负载均衡组件对机顶盒上报的数据进行解析，剥离消息头，仅留下实际有用的消息体。
优选地，在步骤S104，对待处理数据进行分析，获取KPI数据之后，还可以包括以下操作：
步骤S4：对KPI数据进行离线分析挖掘。
在优选实施例中，分布式实时计算***在利用流技术对有效数据进行实时解析与统计，从而得到全网的统计KPI数据之后，可以将KPI数据永久存储至分布式文件***。而离线分析***可以根据需要随时对上述采集的海量数据进行离线分析挖掘。
图2是根据本发明优选实施例的OTT机顶盒数据采集模型的示意图。如图2所示，机顶盒出厂预置第三方应用以及平台方探针组件，同时，预置监控中心上报地址。第三方应用组件主要负责与播放器以及电子节目菜单(EPG)等进行交互，收集用户触发事件、最原始的播放数据等。在国内，牌照方都有具备这些功能的应用组件。探针组件为监控分析***的一部分，可以对牌照方收集的原始数据进行分析加工上报，可以单独进行升级。本发明实施例所提供的技术方案将应用在监控中心中。
在机顶盒放号安装后，运营商可以根据实际情况决定是否通过网管远程修改监控中心上报地址。由于网络规划等各种原因，某些情况下，运营商需要改变分配给监控 中心的相关IP地址。此时，必须及时修改机顶盒的监控中心地址，否则监控中心可能采集不到原始数据。
监控中心的负载均衡组件会进行合法终端的检测，以及网络数据消息头的解析等工作。不合法终端上报的数据将会被丢弃，并记录下该终端基本信息，便于事后排查。分布式实时计算***会启动单独的线程来进行明细数据存储，而不会影响实时的监控分析。
图3是根据本发明优选实施例的基于图2所示的OTT机顶盒数据采集模型的数据处理的流程图。如图3所示，该方法可以包括以下步骤：
第一步、第三方应用组件收集文件下载、用户触发行为以及播放有关的原始数据，并向机顶盒数据采集接口实时传输收集到的原始数据，例如：如果用户已经点击暂停或取消暂停按键，即可将用户点击暂停或取消暂停按键的时间记录下来；如果在播放视频的过程中已经产生了缓冲，则可以将缓冲开始时间以及缓冲时间记录下来。
第二步、机顶盒对采集到的原始数据进行加工计算，按照预设时长(1分钟、30秒甚至10秒)经由协商的数据接口将数据上报至指定的地址。
第三步、采用负载均衡组件来接收机顶盒上报的数据。对机顶盒上报的数据进行解析，剥离消息头，仅留下实际有用的消息体，同时作为分布式消息队列的客户端，将消息体发送至分布式消息队列集群。
第四步、分布式消息队列对接收到的数据进行备份(防止丢失)，同时发送至已经注册监听的分布式的实时计算***处理。
第五步、分布式的实时计算***利用流技术对有效数据进行实时解析与统计，基本上能够实时得到全网的统计关键性能指标(KPI)数据，并还原出用户的观看行为。另外，海量的原始数据也较快地存储至分布式文件***中。
第六步、根据需要择时对上述采集的海量数据进行离线分析挖掘。例如：对片源进行较细分类，对每个用户的观看行为进行分析，发现该用户的兴趣和爱好，从而将最新发布的同类片源推送给该用户；记录用户最近一次观看的节目以及观看到什么位置，下次如果需要继续观看时可以从该位置继续播放。
图4是根据本发明实施例的数据的处理装置的结构框图。如图4所示，该数据的处理装置可以包括：接收模块10，设置为接收机顶盒上报的待处理数据，其中，待处理数据是由机顶盒对从第三方应用组件获取到的原始数据进行加工计算后得到的，原 始数据包括以下至少之一：文件下载数据、用户触发行为数据、文件播放数据；处理模块20，设置为对待处理数据进行分析，获取KPI数据。
采用如图4所示的装置，解决了相关技术中如何利用大数据技术对OTT机顶盒上报的数据进行监控分析的问题，进而可以较为及时地对OTT机顶盒上报的海量原始数据进行统计分析，并且可以实现离线挖掘。
优选地，如图5所示，处理模块20可以包括：解析单元200，设置为采用负载均衡组件对待处理数据进行解析，并将解析后的数据发送至分布式消息队列集群；备份单元202，设置为利用分布式消息队列集群中的分布式消息队列对解析后的数据进行备份，并将解析后的数据发送至分布式实时计算***；分析单元204，设置为通过分布式实时计算***对解析后的数据进行分析统计，得到KPI数据。
优选地，如图5所示，解析单元200可以包括：解析子单元(图中未示出)，设置为采用负载均衡组件从待处理数据中解析出消息头和消息体；处理子单元(图中未示出)，设置为采用负载均衡组件将消息头进行剥离处理，保留消息体。
优选地，接收模块10，设置为按照预设时长经由协商的数据接口接收机顶盒上报的待处理数据。
优选地，如图5所示，上述装置还包括：分析模块30，设置为对KPI数据进行离线分析挖掘。
从以上的描述中，可以看出，上述实施例实现了如下技术效果(需要说明的是这些效果是某些优选实施例可以达到的效果)：本发明实施例提供的技术方案，通过运营商、互联网电视牌照方以及机顶盒厂家三方合作，实现了用户观看行为数据以及效果数据的采集、准实时上报，然后通过大数据技术来对上报的海量数据进行监控分析，从而可以还原用户的观看行为、发现观看中存在的问题以及分析出用户的观看兴趣爱好，以便相关各方及时改进，提高服务质量并有针对性地进行营销，进而可以较为及时地对OTT机顶盒上报的海量原始数据进行统计分析，并且可以实现离线挖掘。另外，由于终端数据的采集过程由运营商、国内主流牌照方以及主流机顶盒厂家共同参与，因此，可以作为行业标准进行推广。
显然，本领域的技术人员应该明白，上述的本发明的各模块或各步骤可以用通用的计算装置来实现，它们可以集中在单个的计算装置上，或者分布在多个计算装置所组成的网络上，可选地，它们可以用计算装置可执行的程序代码来实现，从而，可以将它们存储在存储装置中由计算装置来执行，并且在某些情况下，可以以不同于此处 的顺序执行所示出或描述的步骤，或者将它们分别制作成各个集成电路模块，或者将它们中的多个模块或步骤制作成单个集成电路模块来实现。这样，本发明不限制于任何特定的硬件和软件结合。
以上所述仅为本发明的优选实施例而已，并不用于限制本发明，对于本领域的技术人员来说，本发明可以有各种更改和变化。凡在本发明的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。
工业实用性
如上所述，本发明实施例提供的一种数据的处理方法及装置具有以下有益效果：通过运营商、互联网电视牌照方以及机顶盒厂家三方合作，实现了用户观看行为数据以及效果数据的采集、准实时上报，然后通过大数据技术来对上报的海量数据进行监控分析，从而可以还原用户的观看行为、发现观看中存在的问题以及分析出用户的观看兴趣爱好，以便相关各方及时改进，提高服务质量并有针对性地进行营销，进而可以较为及时地对OTT机顶盒上报的海量原始数据进行统计分析，并且可以实现离线挖掘。

Claims (10)

  1. 一种数据的处理方法，包括：
    接收机顶盒上报的待处理数据，其中，所述待处理数据是由所述机顶盒对从第三方应用组件获取到的原始数据进行加工计算后得到的，所述原始数据包括以下至少之一：文件下载数据、用户触发行为数据、文件播放数据；
    对所述待处理数据进行分析，获取关键性能指标KPI数据。
  2. 根据权利要求1所述的方法，其中，对所述待处理数据进行分析，获取所述KPI数据包括：
    采用负载均衡组件对所述待处理数据进行解析，并将解析后的数据发送至分布式消息队列集群；
    利用所述分布式消息队列集群中的分布式消息队列对所述解析后的数据进行备份，并将所述解析后的数据发送至分布式实时计算***；
    通过所述分布式实时计算***对所述解析后的数据进行分析统计，得到所述KPI数据。
  3. 根据权利要求2所述的方法，其中，采用所述负载均衡组件对所述待处理数据进行解析包括：
    采用所述负载均衡组件从所述待处理数据中解析出消息头和消息体；
    采用所述负载均衡组件将所述消息头进行剥离处理，保留所述消息体。
  4. 根据权利要求1所述的方法，其中，按照预设时长经由协商的数据接口接收所述机顶盒上报的所述待处理数据。
  5. 根据权利要求1所述的方法，其中，在对所述待处理数据进行分析，获取所述KPI数据之后，还包括：
    对所述KPI数据进行离线分析挖掘。
  6. 一种数据的处理装置，包括：
    接收模块，设置为接收机顶盒上报的待处理数据，其中，所述待处理数据是由所述机顶盒对从第三方应用组件获取到的原始数据进行加工计算后得到 的，所述原始数据包括以下至少之一：文件下载数据、用户触发行为数据、文件播放数据；
    处理模块，设置为对所述待处理数据进行分析，获取关键性能指标KPI数据。
  7. 根据权利要求6所述的装置，其中，所述处理模块包括：
    解析单元，设置为采用负载均衡组件对所述待处理数据进行解析，并将解析后的数据发送至分布式消息队列集群；
    备份单元，设置为利用所述分布式消息队列集群中的分布式消息队列对所述解析后的数据进行备份，并将所述解析后的数据发送至分布式实时计算***；
    分析单元，设置为通过所述分布式实时计算***对所述解析后的数据进行分析统计，得到所述KPI数据。
  8. 根据权利要求7所述的装置，其中，所述解析单元包括：
    解析子单元，设置为采用所述负载均衡组件从所述待处理数据中解析出消息头和消息体；
    处理子单元，设置为采用所述负载均衡组件将所述消息头进行剥离处理，保留所述消息体。
  9. 根据权利要求6所述的装置，其中，所述接收模块，设置为按照预设时长经由协商的数据接口接收所述机顶盒上报的所述待处理数据。
  10. 根据权利要求6所述的装置，其中，所述装置还包括：
    分析模块，设置为对所述KPI数据进行离线分析挖掘。
PCT/CN2015/070235 2014-08-11 2015-01-06 数据的处理方法及装置 WO2015154549A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201410393213.5A CN105335437A (zh) 2014-08-11 2014-08-11 数据的处理方法及装置
CN201410393213.5 2014-08-11

Publications (1)

Publication Number Publication Date
WO2015154549A1 true WO2015154549A1 (zh) 2015-10-15

Family

ID=54287271

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2015/070235 WO2015154549A1 (zh) 2014-08-11 2015-01-06 数据的处理方法及装置

Country Status (2)

Country Link
CN (1) CN105335437A (zh)
WO (1) WO2015154549A1 (zh)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10841645B1 (en) 2019-12-09 2020-11-17 Western Digital Technologies, Inc. Storage system and method for video frame segregation to optimize storage
US10897627B2 (en) 2019-01-11 2021-01-19 Western Digital Technologies, Inc. Non-volatile memory system including a partial decoder and event detector for video streams
US11064194B2 (en) 2019-10-31 2021-07-13 Western Digital Technologies, Inc. Encoding digital videos using controllers of data storage devices
US11328511B2 (en) 2020-03-13 2022-05-10 Western Digital Technologies, Inc. Storage system and method for improved playback analysis
US11562018B2 (en) 2020-02-04 2023-01-24 Western Digital Technologies, Inc. Storage system and method for optimized surveillance search

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106028091B (zh) * 2016-05-23 2020-09-22 深圳Tcl数字技术有限公司 视频节目推送方法及装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8027264B2 (en) * 2007-06-01 2011-09-27 Alcatel Lucent Method and apparatus for measuring subjective assessment of digital video impairment
CN102291617A (zh) * 2011-09-03 2011-12-21 四川公用信息产业有限责任公司 Iptv业务端到端故障诊断与定位平台
CN102457390A (zh) * 2010-10-15 2012-05-16 中兴通讯股份有限公司 一种基于qoe的故障定位方法和***
CN102547475A (zh) * 2012-02-21 2012-07-04 德科仕通信(上海)有限公司 提高ip视频媒体流业务质量告警精准度的方法及***
WO2012089708A1 (en) * 2010-12-28 2012-07-05 Telefonica, S.A. Device and method for monitoring end to end traffic and measuring quality delivered to customers in passive optical networks

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101835027B (zh) * 2010-04-16 2012-04-18 中山大学 一种基于双向机顶盒的视频收视率统计及内容推荐***及方法
CN103888798A (zh) * 2012-12-19 2014-06-25 ***通信集团公司 一种用户常用频道之间的切换方法及装置
CN103888810B (zh) * 2014-03-19 2017-11-07 深圳市九洲电器有限公司 一种电视节目排布方法及***
CN103945243A (zh) * 2014-05-16 2014-07-23 无锡北斗星通信息科技有限公司 用于数字电视节目的收看统计设备

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8027264B2 (en) * 2007-06-01 2011-09-27 Alcatel Lucent Method and apparatus for measuring subjective assessment of digital video impairment
CN102457390A (zh) * 2010-10-15 2012-05-16 中兴通讯股份有限公司 一种基于qoe的故障定位方法和***
WO2012089708A1 (en) * 2010-12-28 2012-07-05 Telefonica, S.A. Device and method for monitoring end to end traffic and measuring quality delivered to customers in passive optical networks
CN102291617A (zh) * 2011-09-03 2011-12-21 四川公用信息产业有限责任公司 Iptv业务端到端故障诊断与定位平台
CN102547475A (zh) * 2012-02-21 2012-07-04 德科仕通信(上海)有限公司 提高ip视频媒体流业务质量告警精准度的方法及***

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10897627B2 (en) 2019-01-11 2021-01-19 Western Digital Technologies, Inc. Non-volatile memory system including a partial decoder and event detector for video streams
US11064194B2 (en) 2019-10-31 2021-07-13 Western Digital Technologies, Inc. Encoding digital videos using controllers of data storage devices
US11503285B2 (en) 2019-10-31 2022-11-15 Western Digital Technologies, Inc. Encoding digital videos using controllers of data storage devices
US10841645B1 (en) 2019-12-09 2020-11-17 Western Digital Technologies, Inc. Storage system and method for video frame segregation to optimize storage
US11562018B2 (en) 2020-02-04 2023-01-24 Western Digital Technologies, Inc. Storage system and method for optimized surveillance search
US11328511B2 (en) 2020-03-13 2022-05-10 Western Digital Technologies, Inc. Storage system and method for improved playback analysis

Also Published As

Publication number Publication date
CN105335437A (zh) 2016-02-17

Similar Documents

Publication Publication Date Title
WO2015154549A1 (zh) 数据的处理方法及装置
CN111010614A (zh) 一种显示直播字幕的方法、装置、服务器及介质
EP2615777A1 (en) Monitoring over-the-top adaptive video streaming
US9549010B2 (en) Method and apparatus for media session identification, tracking, and analysis
CN110996109A (zh) 异常直播间清理方法、装置、直播服务器及存储介质
CN105592318B (zh) 一种实现hls频道回看业务的***、设备和方法
AU2014404320B2 (en) Using messaging associated with adaptive bitrate streaming to perform media monitoring for mobile platforms
KR101774983B1 (ko) Ott 비디오의 품질을 모니터링하는 방법, 장치, 및 시스템
US20110153651A1 (en) Apparatus and method for remotely monitoring terminal
CN103974139B (zh) 一种终端网络故障检测方法、设备和***
US20150071620A1 (en) Identifying and rerecording only the incomplete units of a program broadcast recording
CN104539984A (zh) 一种视频播放方法、装置及***
US10405054B2 (en) System and method of remotely determining QoE
US9432731B2 (en) Method and system for detecting live over the top (OTT) streams in communications networks
US20240187325A1 (en) Methods and apparatus to remove information from records
US8204987B2 (en) Providing reports of received multimedia programs
US20230188585A1 (en) Content player performance detection
CN103826158A (zh) 一种网络直播装置和方法
US20130148939A1 (en) System and Method of Recording Media Content
CN103179457A (zh) 一种机顶盒业务交互过程中的故障确定方法和设备
CN112995766B (zh) 用于标识iptv多路视频流的方法和装置
CN108111880B (zh) 排障方法及排障***
EP2139191B1 (en) Method and device for processing data and system comprising such device
US11695584B2 (en) Stateful IGMP fastleave
US10051025B2 (en) Method and apparatus for estimating packet loss

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15777571

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15777571

Country of ref document: EP

Kind code of ref document: A1