WO2022048050A1

WO2022048050A1 - Big data information collection system and usage method

Info

Publication number: WO2022048050A1
Application number: PCT/CN2020/131961
Authority: WO
Inventors: 马樱; 朱顺痣; 卢俊文
Original assignee: 厦门理工学院
Priority date: 2020-09-06
Filing date: 2020-11-26
Publication date: 2022-03-10
Also published as: CN112052149A; CN112052149B

Abstract

A big data information collection system and usage method, comprising a processor, a client terminal, a collector, and various data sensors, the processor being configured to execute instructions of the following operations: determining that data is to be monitored on the client terminal to indicate events; and, in response to said determination: receiving multiple data sets representing multiple inputs or data sensor measurement values collected at a specific client terminal; using a well-trained machine learning model to process the multiple data sets to generate a result corresponding to the predicted risk of an event; and outputting the result on a display device. By means of inputting input data sets comprising communication statistical information into a trained machine learning model and determining that an alarm condition is met on the basis of the result, an alarm can be raised for specific data or transmission rules, effectively ensuring that an alarm can be raised during functions such as data collection or conversion, and thereby enabling effective monitoring of data during transmission and increasing the security of the overall transmission process.

Description

一种大数据信息采集***及使用方法A big data information collection system and using method

技术领域technical field

本发明涉及数据采集技术领域，尤其涉及一种大数据信息采集***及使用方法。The invention relates to the technical field of data collection, in particular to a big data information collection system and a method of using the same.

背景技术Background technique

现阶段工业所使用的传输控制器或传输控制卡及其控制模式，只支持对执行机构进行信号输出或反馈接收的单闭环或开环的操作功能，即在一个命令的执行过程中，数据一旦产生到命令被实现后立刻消失，控制器没有提供保存和分析相关部件的传输数据的机制，不能对机器设备进行故障时间和发生概率的预测，同时也不支持与工业大数据中心的数据交互。At this stage, the transmission controller or transmission control card and its control mode used in the industry only support the single closed-loop or open-loop operation function of signal output or feedback reception to the actuator, that is, during the execution of a command, the data once The controller does not provide a mechanism to save and analyze the transmission data of related components, it cannot predict the failure time and probability of occurrence of machinery and equipment, and it does not support data interaction with industrial big data centers.

如CN109040203A现有技术公开了一种基于大数据的信息采集***，在数据进行远程传输信息和监管服务，虽然降低了人力的投入，但是取法对数据的监管，使得所述数据在传输的过程中极易造成丢失，无法精准应对大量数据的传输的工作。另一种典型的如US20150120777A1的现有技术公开的一种使用触觉反馈挖掘数据的***和方法；和再来看如WO2015061689A1的现有技术公开的一种使用信息反馈挖掘数据的***和方法，传统的设备终端数据传输为单项传输，数据协议复杂，大量带宽被无意义的结构化数据信息占用，同时，有很多无用功浪费在研究与封装上面，还存在终端的数据无法实时采集，无法使得数据无法大量且快速的进行传输，还大量带宽被无意义的结构化数据信息占用。For example, the prior art of CN109040203A discloses an information collection system based on big data. In the process of data remote transmission of information and supervision services, although the input of manpower is reduced, the method of supervision of the data makes the data in the process of transmission. It is very easy to cause loss and cannot accurately cope with the transmission of large amounts of data. Another typical prior art such as US20150120777A1 discloses a system and method for mining data using tactile feedback; The data transmission of the equipment terminal is a single transmission, the data protocol is complex, and a large amount of bandwidth is occupied by meaningless structured data information. At the same time, a lot of useless energy is wasted on research and packaging, and the terminal data cannot be collected in real time. And the transmission is fast, and a large amount of bandwidth is occupied by meaningless structured data information.

为了解决本领域普遍存在无法实现数据的交互、极易造成数据的丢失、无法精准传输、无法处理大量的数据和缺乏监控等等问题，作出了本发明。The present invention is made in order to solve the common problems in the art that data interaction cannot be realized, data loss is easily caused, accurate transmission cannot be performed, a large amount of data cannot be processed, and monitoring is lacking.

技术问题technical problem

本发明的目的在于，针对目前数据采集所存在的不足，提出了一种大数据信息采集***及使用方法。The purpose of the present invention is to propose a big data information collection system and a use method in view of the deficiencies in the current data collection.

技术解决方案technical solutions

为了克服现有技术的不足，本发明采用如下技术方案：In order to overcome the deficiencies of the prior art, the present invention adopts the following technical solutions:

一种大数据信息采集***，包括处理器、客户端、采集器和各个数据传感器，所述处理器配置为执行以下操作的指令：确定要在特定客户端上监视数据以指示事件；并响应于该确定：接收代表在特定客户端处收集的多个输入或数据传感器测量值的多个数据集；使用训练有素的机器学习模型处理多个数据集以生成与事件的预测风险相对应的结果，并在所述显示设备上输出结果。A big data information collection system includes a processor, a client, a collector and various data sensors, the processor is configured to execute instructions for: determining that data is to be monitored on a particular client to indicate an event; and responding to The determination: receiving multiple datasets representing multiple inputs or data sensor measurements collected at a particular client; processing multiple datasets using a trained machine learning model to generate results corresponding to the predicted risk of the event , and output the result on the display device.

可选的，所述多个数据集包括标识特定所述客户端所位于的多个位置的位置数据，处理多个所述数据集包括：标识多个基础数据；特定客户端经常位于的位置区域；使用位置数据确定指示该特定客户端在多个基本位置区域之外的时间变量；并将包括时间变量的输入数据集输入到训练后的机器学习模型，处理多个所述数据集包括：使用所述多个数据集，确定表征与所述特定客户端相关联的最近采集历史或最近采集历史的通信统计；将包括通信统计信息的输入数据集输入到训练后的机器学习模型。Optionally, the multiple data sets include location data identifying multiple locations where the specific client is located, and processing the multiple data sets includes: identifying multiple basic data; location areas where the specific client is often located using the location data to determine a time variable indicating that the particular client is outside of a plurality of base location areas; and inputting an input dataset including the time variable into the trained machine learning model, processing a plurality of said datasets comprising: using The plurality of data sets, determining communication statistics that characterize the most recent acquisition history or recent acquisition history associated with the particular client; inputting the input data set including the communication statistics to the trained machine learning model.

可选的，所述采集器包括网络设备、云计算机、存储器和可执行指令，所述可执行指令存储在所述存储器中，所述处理器被配置为访问至少一个所述存储器并执行所述计算机可执行指令以：接收输入数据；以及至少部分地基于多个分类标准对输入数据进行分类以获得分类的输入数据；至少部分地基于以下至少一项来确定基于云的数据处理功能的第一部分，以从多个云计算机传输到网络设备中：分类的输入数据；多个网络特征；采集网络或网络路径中的多个设备特征；将基于云的数据处理功能的第一部分从多个云计算机传输到网络设备，以在网络设备获得传输的数据处理功能；在第一网络设备上，使用采集的数据处理功能处理输入数据的至少一个子集。Optionally, the collector includes a network device, a cloud computer, a memory, and executable instructions, the executable instructions are stored in the memory, and the processor is configured to access at least one of the memories and execute the Computer-executable instructions to: receive input data; and classify the input data based at least in part on a plurality of classification criteria to obtain classified input data; determine a first portion of the cloud-based data processing function based at least in part on at least one of the following , to transfer from multiple cloud computers to network devices: classified input data; multiple network features; collection of multiple device features in a network or network path; transfer the first part of cloud-based data processing functions from multiple cloud computers transmitting to the network device to obtain the transmitted data processing function at the network device; on the first network device, processing at least a subset of the input data using the collected data processing function.

可选的，所述处理器被配置为通过执行所述计算机可执行指令以确定所述输入数据的量超过阈值来对所述输入数据进行分类，所述处理器被进一步配置为执行计算机可执行指令以：至少部分地响应于确定输入数据的数量超过阈值，使用传输的数据处理功能选择输入数据的一个子集以进行处理；使用基于云的数据处理功能的第二部分，将处理后的输入数据的一个子集发送到多个云计算机中以进行其他数据处理。Optionally, the processor is configured to classify the input data by executing the computer-executable instructions to determine that the amount of the input data exceeds a threshold, the processor is further configured to execute computer-executable instructions the instructions to: in response at least in part to determining that the amount of input data exceeds a threshold, select a subset of the input data for processing using the transmitted data processing function; use a second portion of the cloud-based data processing function to process the processed input A subset of the data is sent to multiple cloud computers for other data processing.

可选的，所述处理器还被配置为执行所述计算机可执行指令以：确定网络等待时间超过阈值等待时间；以及部分地响应于确定网络等待时间超过阈值等待时间，使用传输的数据处理功能选择输入数据的一个子集以进行处理；使用基于云的数据处理功能的第二部分，将处理后的输入数据的至少一个子集发送到多个云计算机中以进行数据的处理。Optionally, the processor is further configured to execute the computer-executable instructions to: determine that the network latency exceeds a threshold latency; and use the transmitted data processing function in part in response to determining that the network latency exceeds the threshold latency A subset of the input data is selected for processing; using the second part of the cloud-based data processing function, at least a subset of the processed input data is sent to a plurality of cloud computers for processing of the data.

另外，本发明提供一种大数据信息使用方法，所述使用方法包括：使用多个所述数据集，确定表征与特定客户端相关联的最近采集历史或最近采集历史的通信统计；将包括通信统计信息的输入数据集输入到训练后的机器学习模型；所述通信统计量包括：多个采集或多个外发采集的计数；以及多个采集的持续时间统计；多个外发采集的长度统计；与多个采集或多个传出采集正在与之通信的多个不同数据采集有关的子集多样性统计；多个所述数据集包括表征所述特定客户端的各个所述数据传感器的数据，处理多个所述数据集包括：基于所述数据传感器测量值，生成传输统计量代表特定客户端的持续时间、强度或移动频率；将包括通信统计信息的输入数据集输入到训练后的机器学习模型，基于结果确定满足警报条件；以及作为确定满足警报条件的结果，将结果发送到另一客户端。In addition, the present invention provides a method of using big data information, said method of using comprising: using a plurality of said data sets, determining communication statistics characterizing recent acquisition history or recent acquisition history associated with a particular client; including communication An input dataset of statistics is input to the trained machine learning model; the communication statistics include: counts of multiple acquisitions or multiple outgoing acquisitions; and statistics of the duration of multiple acquisitions; length of multiple outgoing acquisitions Statistics; subset diversity statistics related to multiple acquisitions or multiple different data acquisitions with which multiple outgoing acquisitions are communicating; a plurality of said data sets including data characterizing each of said data sensors of said particular client , processing a plurality of said data sets comprising: based on said data sensor measurements, generating transmission statistics representing the duration, intensity or frequency of movement of a particular client; inputting an input data set including communication statistics into a trained machine learning The model determines that the alert condition is satisfied based on the result; and sends the result to another client as a result of determining that the alert condition is satisfied.

可选的，所述使用方法还包括：所述客户端基于多个用户的数据来访问用户分类模型；以及其中确定要在特定客户端处监视数据以预测事件的步骤包括：由另一客户端使用用户分类模型将用户分类为用户组；另一客户端至少部分地基于对用户进行分类的用户组来选择训练有素的机器学习模型，由另一所述客户端在多个所述用户的数据的至少一部分上执行功能以产生组表达；以及所述另一客户端将所述组表达式映射到N维空间；所述另一客户端将所述组表达式所包围的区域分类为用户组。Optionally, the method of using further includes: the client accessing a user classification model based on data of a plurality of users; and wherein the step of determining that data is to be monitored at a particular client to predict events includes: by another client Using a user classification model to classify users into user groups; another client selects a trained machine learning model based at least in part on the user groups classifying the users, and another said client selects a trained machine learning model among a plurality of said users. performing a function on at least a portion of the data to generate a group expression; and the another client maps the group expression to an N-dimensional space; the another client classifies the area enclosed by the group expression as a user Group.

有益效果beneficial effect

1. 通过将输入数据分为针对实时数据流的第一部分和针对非实时数据流的第二部分表示在网络上执行的多路径处理的形式，保证数据在传输或者采集的过程中，能够进行大量数据采集和存储，提高整个***对大量数据的高效的采集。1. By dividing the input data into the first part for the real-time data stream and the second part for the non-real-time data stream in the form of multi-path processing performed on the network, it is ensured that the data can be processed in a large amount during the process of transmission or collection. Data collection and storage, improve the efficient collection of large amounts of data by the entire system.

2. 通过确定输入数据的量超过阈值，阀值包括数据最多的阀值量，则这触发基于云的功能向存储器的迁移以及对基于云的功能的选择，输入数据的某些部分，使用迁移的功能在存储器进行处理。2. By determining that the amount of input data exceeds a threshold, the threshold includes the threshold amount with the most data, then this triggers the migration of cloud-based functions to storage and the selection of cloud-based functions, certain parts of the input data, using migration The functions are processed in memory.

3. 通过在网络高延迟的情况下，使用迁移功能在存储器处理的输入数据可能是需要实时分析的数据，在采集网络中，执行迁移的功能的存储器设备可能比云计算机更靠近数据源，从而消除了如果将数据直接发送给高延迟会导致的处理延迟，由从采集网络迁移到存储器的功能执行的处理包括数据清理、数据过滤、数据标准化、数据转换、数据汇总、数据分析或任何其他合适形式的数据处理。3. In the case of high network latency, the input data processed in the memory using the migration function may be data that needs to be analyzed in real time. In the acquisition network, the memory device that performs the migration function may be closer to the data source than the cloud computer. Eliminates the processing delays that would result if data were sent directly to high latency, processing performed by functions migrating from the acquisition network to storage including data cleaning, data filtering, data normalization, data transformation, data aggregation, data analysis or any other suitable form of data processing.

4.通过将包括通信统计信息的输入数据集输入到训练后的机器学习模型，并基于结果确定满足警报条件，使得使用者能够根据报警或者数据的采集或者传输的过程进行采集，对特定数据或者传输规则进行报警，有效保证数据进行采集或者转换等功能时能够报警，实现对数据在传输的过程中能够有效的进行监管，提高整个传输过程的安全性。4. By inputting the input data set including communication statistics into the trained machine learning model, and determining that the alarm conditions are satisfied based on the results, the user can collect according to the process of alarm or data collection or transmission, and the specific data or The transmission rules are used for alarming, which can effectively ensure that the data can be alarmed when the data is collected or converted, so that the data can be effectively supervised during the transmission process, and the security of the entire transmission process can be improved.

5.通过采用客户端在多个用户的数据的至少一部分上执行功能以产生组表达，保证各个客户端在进行数据传输或者采集的过程中，能够对各个客户端进行分配保证在进行采集的过程中，不会存在数据之间的传输进行相互的干扰，有效的提高与采集器进行连接的各个客户端的数据交互的高效。5. By using the client to perform functions on at least a part of the data of multiple users to generate group expressions, it is ensured that each client can be allocated to each client during the process of data transmission or collection to ensure that the process of collection is performed. There is no mutual interference in the transmission of data, which effectively improves the efficiency of data interaction of each client connected to the collector.

附图说明Description of drawings

从以下结合附图的描述可以进一步理解本发明。图中的部件不一定按比例绘制，而是将重点放在示出实施例的原理上。在不同的视图中，相同的附图标记指定对应的部分。The present invention can be further understood from the following description in conjunction with the accompanying drawings. The components in the figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the embodiments. Like reference numerals designate corresponding parts throughout the different views.

图1为所述采集装置的结构示意图。FIG. 1 is a schematic structural diagram of the collection device.

图2为所述采集***的应用场景示意图。FIG. 2 is a schematic diagram of an application scenario of the acquisition system.

图3为所述采集***的控制流程示意图。FIG. 3 is a schematic diagram of the control flow of the acquisition system.

图4为采集数据的控制流程示意图。FIG. 4 is a schematic diagram of a control flow for collecting data.

图5为所述采集器的控制流程示意图。FIG. 5 is a schematic diagram of the control flow of the collector.

图6为所述处理器处理数据的控制流程示意图。FIG. 6 is a schematic diagram of a control flow for processing data by the processor.

图7为所述采集网络的控制流程示意图。FIG. 7 is a schematic diagram of a control flow of the acquisition network.

附图标号说明：1-显示装置；2-输出端口；3-控住面板；4-采集端口；5-采集装置本体。Description of reference numerals: 1-display device; 2-output port; 3-control panel; 4-collection port; 5-collection device body.

本发明的最佳实施方式BEST MODE FOR CARRYING OUT THE INVENTION

为了使得本发明的目的.技术方案及优点更加清楚明白，以下结合其实施例，对本发明进行进一步详细说明；应当理解，此处所描述的具体实施例仅用于解释本发明，并不用于限定本发明。对于本领域技术人员而言，在查阅以下详细描述之后，本实施例的其它***.方法和/或特征将变得显而易见。旨在所有此类附加的***.方法.特征和优点都包括在本说明书内.包括在本发明的范围内，并且受所附权利要求书的保护。在以下详细描述描述了所公开的实施例的另外的特征，并且这些特征根据以下将详细描述将是显而易见的。In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be described in further detail below in conjunction with its embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention, not to limit the present invention. invention. Other systems, methods and/or features of the present embodiments will become apparent to those skilled in the art upon review of the following detailed description. It is intended that all such additional systems, methods, features and advantages be included within this description, be within the scope of the invention, and be protected by the accompanying claims. Additional features of the disclosed embodiments are described in the following detailed description and will be apparent from the following detailed description.

本发明实施例的附图中相同或相似的标号对应相同或相似的部件；在本发明的描述中，需要理解的是，若有术语“上”.“下”.“左”.“右”等指示的方位或位置关系为基于附图所示的方位或位置关系，仅是为了便于描述本发明和简化描述，而不是指示或暗示所指的装置或组件必须具有特定的方位.以特定的方位构造和操作，因此附图中描述位置关系的用语仅用于示例性说明，不能理解为对本专利的限制，对于本领域的普通技术人员而言，可以根据具体情况理解上述术语的具体含义。The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar components; in the description of the present invention, it should be understood that if the terms “upper”, “lower”, “left” and “right” are used The orientation or positional relationship indicated by etc. is based on the orientation or positional relationship shown in the accompanying drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred device or component must have a specific orientation. Orientation structure and operation, so the terms describing the positional relationship in the accompanying drawings are only used for exemplary illustration, and should not be construed as a limitation on the present patent. Those of ordinary skill in the art can understand the specific meanings of the above terms according to specific situations.

实施例一：一种大数据信息采集***，包括处理器、客户端、采集器和各个数据传感器，所述处理器配置为执行以下操作的指令：确定要在特定客户端上监视数据以指示事件；并响应于该确定：接收代表在特定客户端处收集的多个输入或数据传感器测量值的多个数据集；使用训练有素的机器学习模型处理多个数据集以生成与事件的预测风险相对应的结果，并在所述显示设备上输出结果；所述多个数据集包括标识特定所述客户端所位于的多个位置的位置数据，处理多个所述数据集包括：标识多个基础数据；特定客户端经常位于的位置区域；使用位置数据确定指示该特定客户端在多个基本位置区域之外的时间变量；并将包括时间变量的输入数据集输入到训练后的机器学习模型，处理多个所述数据集包括：使用所述多个数据集，确定表征与所述特定客户端相关联的最近采集历史或最近采集历史的通信统计；将包括通信统计信息的输入数据集输入到训练后的机器学习模型；所述采集器包括网络设备、云计算机、存储器和可执行指令，所述可执行指令存储在所述存储器中，所述处理器被配置为访问至少一个所述存储器并执行所述计算机可执行指令以：接收输入数据；以及至少部分地基于多个分类标准对输入数据进行分类以获得分类的输入数据；至少部分地基于以下至少一项来确定基于云的数据处理功能的第一部分，以从多个云计算机传输到网络设备中：分类的输入数据；多个网络特征；采集网络或网络路径中的多个设备特征；将基于云的数据处理功能的第一部分从多个云计算机传输到网络设备，以在网络设备获得传输的数据处理功能；在第一网络设备上，使用采集的数据处理功能处理输入数据的至少一个子集；所述处理器被配置为通过执行所述计算机可执行指令以确定所述输入数据的量超过阈值来对所述输入数据进行分类，所述处理器被进一步配置为执行计算机可执行指令以：至少部分地响应于确定输入数据的数量超过阈值，使用传输的数据处理功能选择输入数据的一个子集以进行处理；使用基于云的数据处理功能的第二部分，将处理后的输入数据的一个子集发送到多个云计算机中以进行其他数据处理；所述处理器还被配置为执行所述计算机可执行指令以：确定网络等待时间超过阈值等待时间；以及部分地响应于确定网络等待时间超过阈值等待时间，使用传输的数据处理功能选择输入数据的一个子集以进行处理；使用基于云的数据处理功能的第二部分，将处理后的输入数据的至少一个子集发送到多个云计算机中以进行数据的处理；所述使用方法包括：使用多个所述数据集，确定表征与特定客户端相关联的最近采集历史或最近采集历史的通信统计；将包括通信统计信息的输入数据集输入到训练后的机器学习模型；所述通信统计量包括：多个采集或多个外发采集的计数；以及多个采集的持续时间统计；多个外发采集的长度统计；与多个采集或多个传出采集正在与之通信的多个不同数据采集有关的子集多样性统计；多个所述数据集包括表征所述特定客户端的各个所述数据传感器的数据，处理多个所述数据集包括：基于所述数据传感器测量值，生成传输统计量代表特定客户端的持续时间、强度或移动频率；将包括通信统计信息的输入数据集输入到训练后的机器学习模型，基于结果确定满足警报条件；以及作为确定满足警报条件的结果，将结果发送到另一客户端；所述使用方法还包括：所述客户端基于多个用户的数据来访问用户分类模型；以及其中确定要在特定客户端处监视数据以预测事件的步骤包括：由另一客户端使用用户分类模型将用户分类为用户组；另一客户端至少部分地基于对用户进行分类的用户组来选择训练有素的机器学习模型，由另一所述客户端在多个所述用户的数据的至少一部分上执行功能以产生组表达；以及所述另一客户端将所述组表达式映射到N维空间；所述另一客户端将所述组表达式所包围的区域分类为用户组。Embodiment 1: A system for collecting big data information, including a processor, a client, a collector and various data sensors, the processor is configured to execute instructions for the following operations: determine that data is to be monitored on a specific client to indicate an event ; and in response to the determination: receive multiple datasets representing multiple inputs or data sensor measurements collected at a particular client; process the multiple datasets using a trained machine learning model to generate a predicted risk associated with the event corresponding results, and output the results on the display device; the plurality of data sets include location data identifying a plurality of locations where the specific client is located, and processing a plurality of the data sets includes: identifying a plurality of Base data; location areas in which a particular client is frequently located; use location data to determine time variables that indicate that particular client is outside of a number of base location areas; and input an input dataset including time variables into a trained machine learning model , processing a plurality of the data sets comprising: using the plurality of data sets, determining communication statistics characterizing the most recent acquisition history or recent acquisition history associated with the particular client; inputting the input data set including the communication statistics information to the trained machine learning model; the collector includes a network device, a cloud computer, a memory, and executable instructions, the executable instructions are stored in the memory, and the processor is configured to access at least one of the memories and execute the computer-executable instructions to: receive input data; and classify the input data based at least in part on a plurality of classification criteria to obtain classified input data; determine cloud-based data processing based at least in part on at least one of the following The first part of the function to transfer from multiple cloud computers into network devices: classified input data; multiple network characteristics; collection of multiple device characteristics in the network or network path; the first part of the cloud-based data processing function from a plurality of cloud computers are transmitted to the network device to obtain the transmitted data processing functions at the network device; on the first network device, at least a subset of the input data is processed using the collected data processing functions; the processor is configured to pass Executing the computer-executable instructions to determine that the amount of the input data exceeds a threshold to classify the input data, the processor is further configured to execute the computer-executable instructions to: Quantity exceeds a threshold, use the transmitted data processing function to select a subset of the input data for processing; use the second part of the cloud-based data processing function to send a subset of the processed input data to multiple cloud computers for other data processing; the processor is further configured to execute the computer-executable instructions to: determine that the network latency exceeds a threshold latency; and use the transmitted data in part in response to determining that the network latency exceeds the threshold latency The processing function selects a subset of the input data for processing; using the second part of the cloud-based data processing function, sends at least a subset of the processed input data to a plurality of cloud computers for processing of the data; and envoy The method includes: using a plurality of said data sets, determining communication statistics characterizing recent acquisition history or recent acquisition history associated with a particular client; inputting an input data set including communication statistics into a trained machine learning model; The communication statistics include: counts of multiple acquisitions or multiple outgoing acquisitions; and duration statistics of multiple acquisitions; length statistics of multiple outgoing acquisitions; Subset diversity statistics related to a plurality of different data collections of communication; a plurality of said data sets including data characterizing each of said data sensors of said particular client, and processing a plurality of said data sets includes: based on said data sensors measurements, generating transmission statistics representing the duration, intensity, or frequency of movement of a particular client; inputting an input dataset including communication statistics into a trained machine learning model, determining based on the results that an alert condition is met; and as determining that an alert condition is met and sending the result to another client; the method of use further comprising: the client accessing a user classification model based on data of a plurality of users; The steps include: using a user classification model by another client to classify users into user groups; another client selecting a trained machine learning model based at least in part on the user groups classifying the users, and the other client a client performs a function on at least a portion of a plurality of the users' data to generate a group expression; and the another client maps the group expression to an N-dimensional space; the another client maps the group expression The area enclosed by the formula is classified as a user group.

实施例二：本实施例应当理解为至少包含前述任一一个实施例的全部特征，并在其基础上进一步改进，具体的，提供一种大数据信息采集***，包括处理器、客户端、采集器和各个数据传感器，所述处理器配置为执行以下操作的指令：确定要在特定客户端上监视数据以指示事件；并响应于该确定：接收代表在特定客户端处收集的多个输入或数据传感器测量值的多个数据集；使用训练有素的机器学习模型处理多个数据集以生成与事件的预测风险相对应的结果，并在所述显示设备上输出结果；具体的，所述采集器和各个所述数据采集传感器之间对所述数据进行采集的操作，保证所述数据在被采集的过程中，能够对大量所述数据进行收集并就通过所述客户端进行分发的操作；在本实施例中，所述客户端为数据终端，所述数据终端被用来对所述数据进行采集和传输的，另外，所述数据在进行采集的过程中，通过所述采集器与各个所述数据传感器进行配合使用，保证所述数据采集或者传输的过程中能够对整个采集装置的采集的过程高效的采集；在本实施例中，所述处理器对所述可执行指令进行运行，并完成在特定客户端上监视数据以指示事件，同时基于确定后的相应采集所述数据集或者接收所述数据传感器的数据集；在本实施例中，还通过利用所述机器学习模型对一个或者多个数据集进行风险的预测，并把预测的结果在显示设备上进行显示；在本实施例中，所述机器学习模型被配置为学习过的模型，能够对所述数据集进行采集或进行与数据有关的数据采集工作。Embodiment 2: This embodiment should be understood to include at least all the features of any one of the foregoing embodiments, and further improve on the basis thereof. Specifically, a big data information collection system is provided, including a processor, a client, a a collector and respective data sensors, the processor configured to execute instructions to: determine that data is to be monitored at the particular client to indicate an event; and in response to the determination: receive a plurality of inputs representative of the collection at the particular client or multiple datasets of data sensor measurements; processing multiple datasets using a trained machine learning model to generate results corresponding to the predicted risk of an event, and outputting the results on said display device; specifically, so The operation of collecting the data between the collector and each of the data collection sensors ensures that a large amount of the data can be collected and distributed through the client during the process of being collected. operation; in this embodiment, the client is a data terminal, and the data terminal is used to collect and transmit the data. In addition, in the process of collecting the data, the data is collected by the collector. It is used in conjunction with each of the data sensors to ensure that the data collection or transmission process can efficiently collect the collection process of the entire collection device; in this embodiment, the processor executes the execution of the executable instructions. run, and complete monitoring data on a specific client to indicate an event, and at the same time collect the data set or receive the data set of the data sensor based on the determined corresponding; in this embodiment, also by using the machine learning model Perform risk prediction on one or more data sets, and display the predicted results on a display device; in this embodiment, the machine learning model is configured as a learned model, which can perform Collect or perform data collection work related to data.

所述多个数据集包括标识特定所述客户端所位于的多个位置的位置数据，处理多个所述数据集包括：标识多个基础数据；特定客户端经常位于的位置区域；使用位置数据确定指示该特定客户端在多个基本位置区域之外的时间变量；并将包括时间变量的输入数据集输入到训练后的机器学习模型，处理多个所述数据集包括：使用所述多个数据集，确定表征与所述特定客户端相关联的最近采集历史或最近采集历史的通信统计；将包括通信统计信息的输入数据集输入到训练后的机器学习模型；具体的，收集或者采集的数据通过应用程序的执行被处理以生成可预测用户是否正在经历事件的结果；在某些情况下，结果对应于推断采集过程或对用户的一个或多个大数据***的评估，这能够预测采集的程度；所收集数据的处理扩展到使用简单或多个线性回归模型之外；更复杂的时间序列分析或统计学习技术确定通过简单特征无法检测到的预测特征；以前尚未报告过基于通过电子设备收集的被动监视数据估算结果的结果；这种方法改善监测工作并进一步了解事件的生理和行为决定因素以及与这些事件相关的因素；在本实施例中，基于被动监视的所述数据传感器数据来构造特定于受到采集的数据集的模型以估计预测的结果；所述特定的客户端在一段时间内被动地收集传感器数据，并且在一段时间内在一个或多个时间点收集输入；另外，还能够通过大量的训练所述机器学习模型以学习可用于将传感器数据转换为数据采集的高效的进行；在某些情况下，机器学习模型的输出包括或经过处理以生成分类值，本本实例中，处理采集到的数据输出以将打包或者加密的节点的方式分配给集群；在某些情况下，提供了仅基于经由电子设备收集的被动数据来推断事件，即：根据采集的被动数据进行分析进行传输过程或加密过程，在本实施例中，所述被动数据包括不经过所述电子设备进行加工的数据；随着个人手持通信技术的迅速发展，在本实施例中还提供一中用于对手持客户端进行数据采集方法，所述方法包括从这些设备收集的数据提供高度合规的信号，从而深入了解位置和通信数据都可能对所述，并且通过电子设备收集的这些数据可用于评估数据传输的质量；所收集和存储的数据作为输入传递到机器学习模型，以产生预测数据采集或者传输的风险结果；所述结果显示在电子设备或远程服务器上；所述机器学习模型执行的结果被处理以确定是否满足警报条件，执行如警报条件指示当结果高于预定阈值时将要呈现警报；对警报条件可能还会触发设备操作的更改；改变所述数据传感器输出的被动监视采集数据的频率或改变所述数据传感器采集所述数据侧的频率； The plurality of data sets include location data identifying a plurality of locations where the specific client is located, and processing the plurality of the data sets includes: identifying a plurality of underlying data; location areas where the specific client is often located; using the location data determining a time variable indicating that the particular client is outside of a plurality of base location areas; and inputting an input dataset including the time variable into the trained machine learning model, processing a plurality of the data sets comprising: using the plurality of Data set, determine the communication statistics that characterize the most recent acquisition history or the most recent acquisition history associated with the specific client; input the input data set including the communication statistical information into the trained machine learning model; specifically, the collected or collected The data is processed through the execution of the application to generate results that predict whether the user is experiencing an event; in some cases, the results correspond to inferring the acquisition process or evaluation of one or more of the user's big data systems, which are capable of predicting the acquisition extent; processing of collected data extends beyond the use of simple or multiple linear regression models; more sophisticated time series analysis or statistical learning techniques to identify predictive features not detectable by simple features; The result of the evaluation of the results of the passive surveillance data collected; this approach improves surveillance and further understanding of the physiological and behavioral determinants of events and the factors associated with those events; in this example, based on the passive surveillance data sensor data constructing a model specific to the collected dataset to estimate the outcome of the prediction; the specific client passively collects sensor data over a period of time and inputs at one or more points in time over a period of time; additionally, is able to The machine learning model is extensively trained to learn efficient processes that can be used to convert sensor data into data collection; in some cases, the output of the machine learning model includes or is processed to generate categorical values, in this example, processing The collected data output is distributed to the cluster in the form of packaged or encrypted nodes; in some cases, inferring events based only on passive data collected via electronic devices is provided, i.e., analysis based on collected passive data for transmission process or encryption process, in this embodiment, the passive data includes data that is not processed by the electronic device; with the rapid development of personal handheld communication technology, this embodiment also provides a Client-side data collection methods that include data collected from these devices providing a highly compliant signal, providing insight into both location and communication data that may be of interest to the described, and that data collected through electronic devices can be used to assess data transmissions the quality of data collected and stored; the data collected and stored is passed as input to a machine learning model to produce results that predict the risk of data collection or transmission; the results are displayed on an electronic device or remote server; the results of the execution of the machine learning model are processed To determine if an alarm condition is met, do as the alarm condition indicates that an alarm is to be presented when the result is above a predetermined threshold; a device action may also be triggered for an alarm condition Changes made; change the frequency of passive monitoring of the data sensor output to collect data or change the frequency of the data sensor to collect the data side;

所述采集器包括网络设备、云计算机、存储器和可执行指令，所述可执行指令存储在所述存储器中，所述处理器被配置为访问至少一个所述存储器并执行所述计算机可执行指令以：接收输入数据；以及至少部分地基于多个分类标准对输入数据进行分类以获得分类的输入数据；至少部分地基于以下至少一项来确定基于云的数据处理功能的第一部分，从多个云计算机传输到网络设备中：分类的输入数据；多个网络特征；采集网络或网络路径中的多个设备特征；将基于云的数据处理功能的第一部分从多个云计算机传输到网络设备，以在网络设备获得传输的数据处理功能；在第一网络设备上，使用采集的数据处理功能处理输入数据的至少一个子集；具体的，在本实施例中，在数据进行采集或者收集的过程中，需要对所述各个所述采集器进行采集网络的搭建，所述云网络包括云计算机，并从所述云网络中进行以下之一的采集操作：i）分类的输入数据；多个网络特征；iii）采集网络中或沿网络路径的多个设备特征，并将基于云的数据处理功能的第一部分从多个云计算机传输到网络设备，以在网络设备获得传输的数据处理功能；在第一网络设备上，使用采集的数据处理功能处理输入数据的至少一个子集；The collector includes a network device, a cloud computer, a memory, and executable instructions, the executable instructions being stored in the memory, the processor configured to access at least one of the memories and execute the computer-executable instructions to: receive input data; and classify the input data based at least in part on a plurality of classification criteria to obtain classified input data; determine a first portion of the cloud-based data processing function based, at least in part on at least one of the following, from a plurality of Cloud computer transmission into network devices: classified input data; multiple network features; collection of multiple device characteristics in a network or network path; transmission of the first part of cloud-based data processing functions from multiple cloud computers to network devices, Obtain the transmitted data processing function on the network device; on the first network device, use the collected data processing function to process at least a subset of the input data; specifically, in this embodiment, in the process of data collection or collection , it is necessary to construct a collection network for each of the collectors, the cloud network includes a cloud computer, and one of the following collection operations is performed from the cloud network: i) classified input data; multiple networks characteristics; iii) collect characteristics of multiple devices in the network or along network paths, and transmit the first portion of cloud-based data processing functions from multiple cloud computers to the network equipment to obtain the transmitted data processing functions at the network equipment; On the first network device, use the collected data processing function to process at least a subset of the input data;

所述处理器被配置为通过执行所述计算机可执行指令以确定所述输入数据的量超过阈值来对所述输入数据进行分类，所述处理器被进一步配置为执行计算机可执行指令以：至少部分地响应于确定输入数据的数量超过阈值，使用传输的数据处理功能选择输入数据的一个子集以进行处理；使用基于云的数据处理功能的第二部分，将处理后的输入数据的一个子集发送到多个云计算机中以进行其他数据处理；具体的，在本实施例中，用于对输入数据进行分类的分类标准包括输入数据的大小或者数量，对输入数据的实时处理的需求等等；如果确定输入数据的量超过阈值，所述阀值包括数据最多的阀值量，则这触发基于云的功能向存储器的迁移以及对基于云的功能的选择；输入数据的某些部分，使用迁移的功能在存储器进行处理；以这种方式，减少发送到采集网络以进行额外处理的数据量；在本实施例中，还与在存储器处对输入数据的某些部分的处理并行地将全部原始输入数据以潜在地压缩并成批地发送到采集网络；根据分类标准对输入数据进行分类包括评估输入数据的任何部分是否需要实时处理；输入数据的第一部分形成实时数据流的一部分，根据该实时数据流，使用迁移到边缘的功能来处理输入数据的第一部分，以便将实时响应提供给特定客户端；输入数据的第二部分形成非实时数据流的一部分，所述非实时数据流也选择使用迁移的功能进行处理，但可能并不表示需要实时响应，因此进行压缩和压缩；批量发送到采集网络以进行其他处理；输入数据的第二部分包括输入数据的整个第一部分，输入数据的第一部分的子集或与第一部分完全不同的一组输入数据；实时数据流对应于需要执行某种形式的实时分析的输入数据；相反，形成非实时数据流的一部分的输入数据可能不需要立即分析，而是包括与监管要求有关的数据、异步数据、需要近期或长期分析的数据等；将输入数据分为针对实时数据流的第一部分和针对非实时数据流的第二部分表示在网络上执行的多路径处理的形式，保证所述数据在传输或者采集的过程中，能够进行大量数据采集和存储，提高整个***对大量数据的高效的采集；The processor is configured to classify the input data by executing the computer-executable instructions to determine that the amount of the input data exceeds a threshold, the processor is further configured to execute the computer-executable instructions to: at least In part in response to determining that the amount of input data exceeds a threshold, a subset of the input data is selected for processing using the transmitted data processing function; using a second portion of the cloud-based data processing function, a subset of the processed input data is processed The data set is sent to multiple cloud computers for other data processing; specifically, in this embodiment, the classification criteria used to classify the input data include the size or quantity of the input data, the requirements for real-time processing of the input data, etc. etc.; if it is determined that the amount of input data exceeds a threshold including the threshold amount of the most data, then this triggers the migration of cloud-based functions to storage and the selection of cloud-based functions; some portion of the input data, The processing is performed in memory using the functionality of migration; in this way, the amount of data sent to the acquisition network for additional processing is reduced; in this embodiment, also in parallel with the processing of some parts of the input data at memory All raw input data to potentially be compressed and sent in batches to the acquisition network; classifying the input data according to classification criteria includes evaluating whether any part of the input data requires real-time processing; the first part of the input data forms part of the real-time data stream, according to This real-time data stream, uses the functionality migrated to the edge to process the first part of the input data in order to provide a real-time response to a specific client; the second part of the input data forms part of the non-real-time data stream, which also Select to use migrated features for processing, but may not indicate a need for real-time response, so compress and compress; send in batches to the acquisition network for additional processing; the second part of the input data includes the entire first part of the input data, the A subset of the first part or a completely different set of input data from the first part; a real-time data stream corresponds to input data that needs to perform some form of real-time analysis; in contrast, input data that forms part of a non-real-time data stream may not require immediate Analysis, but instead includes data related to regulatory requirements, asynchronous data, data requiring near-term or long-term analysis, etc.; split input data into a first part for real-time data streams and a second part for non-real-time data streams represented on the network The form of multi-path processing performed ensures that a large amount of data can be collected and stored during the process of data transmission or collection, thereby improving the efficient collection of large amounts of data by the entire system;

所述处理器还被配置为执行所述计算机可执行指令以：确定网络等待时间超过阈值等待时间；以及部分地响应于确定网络等待时间超过阈值等待时间，使用传输的数据处理功能选择输入数据的一个子集以进行处理；使用基于云的数据处理功能的第二部分，将处理后的输入数据的至少一个子集发送到多个云计算机中以进行数据的处理；具体的，在本实施例中，确定网络等待时间的网络特征影响功能是否从采集网络迁移到存储器；如果网络等待时间特别高，如：超过阈值等待时间，则将基于云的功能的某些部分迁移到存储器，以允许至少一部分输入数据绕过高网络等待时间并使用迁移的功能在存储器进行处理；在网络高延迟的情况下，使用迁移功能在存储器处理的输入数据可能是需要实时分析的数据；在采集网络中，执行迁移的功能的存储器设备可能比云计算机更靠近数据源，从而消除了如果将数据直接发送给高延迟会导致的处理延迟；由从采集网络迁移到存储器的功能执行的处理包括数据清理、数据过滤、数据标准化、数据转换、数据汇总、数据分析或任何其他合适形式的数据处理；此外，从采集网络迁移到存储器的功能的类型是上下文或应用特定的；提供实时数据的短期分析的功能迁移到存储器，而提供更多长期分析的其他功能则保留在采集网络中；基于云的数据处理功能可包括数据分析数据过滤对过滤后的数据转换等；使用云数据库内的另一数据源对转换后的数据进行分析，出于审计目的而存储原始数据，并存储分析结果以立即使用；如果要考虑将大量数据传输到云或由于网络带宽而导致网络延迟，则将一些基于云的功能迁移到存储器，或更具体地说，迁移到沿任何地方的一个或多个设备网络路径，比如：过滤功能迁移到存储器；The processor is also configured to execute the computer-executable instructions to: determine that the network latency exceeds a threshold latency; and in part in response to determining that the network latency exceeds the threshold latency, select a data processing function of the input data using the transmitted data processing function. a subset for processing; using the second part of the cloud-based data processing function, at least a subset of the processed input data is sent to multiple cloud computers for data processing; specifically, in this embodiment , determine whether network characteristics of network latency affect the migration of functions from the acquisition network to storage; if network latency is particularly high, such as exceeding a threshold latency, migrate some parts of cloud-based functions to storage to allow at least Part of the input data bypasses high network latency and is processed in memory using the migration function; in the case of high network latency, the input data processed in memory using the migration function may be data that needs to be analyzed in real time; in the acquisition network, execute Storage devices for migrated functions may be closer to the data source than cloud computers, eliminating processing delays that would result if data were sent directly to high latency; processing performed by functions migrated from acquisition network to storage includes data cleaning, data filtering , data normalization, data transformation, data aggregation, data analysis, or any other suitable form of data processing; in addition, the type of functionality migrated from the acquisition network to storage is context- or application-specific; functionality that provides short-term analysis of real-time data migrates to storage, while other functions that provide more long-term analysis remain in the acquisition network; cloud-based data processing functions may include data analysis, data filtering, filtered data transformation, etc.; using another data source within the cloud database to data for analysis, store raw data for auditing purposes, and store analysis results for immediate use; migrate some cloud-based functions to storage if large amounts of data are to be transferred to the cloud or network latency due to network bandwidth is a concern , or more specifically, migrate to one or more devices anywhere along the network path, e.g. filter functions migrate to storage;

所述使用方法包括：使用多个所述数据集，确定表征与特定客户端相关联的最近采集历史或最近采集历史的通信统计；将包括通信统计信息的输入数据集输入到训练后的机器学习模型；所述通信统计量包括：多个采集或多个外发采集的计数；以及多个采集的持续时间统计；多个外发采集的长度统计；与多个采集或多个传出采集正在与之通信的多个不同数据采集有关的子集多样性统计；多个所述数据集包括表征所述特定客户端的各个所述数据传感器的数据，处理多个所述数据集包括：基于所述数据传感器测量值，生成传输统计量代表特定客户端的持续时间、强度或移动频率；将包括通信统计信息的输入数据集输入到训练后的机器学习模型，基于结果确定满足警报条件；以及作为确定满足警报条件的结果，将结果发送到另一客户端；使得使用者能够根据报警或者所述数据的采集或者传输的过程进行采集，另外，在实施例中，通过对特定数据或者传输规则进行报警，有效保证所述数据进行采集或者转换等功能时能够报警，实现对所述数据在传输的过程中能够有效的进行监管，提高整个传输过程的安全性；The method of use includes: using a plurality of the data sets, determining communication statistics that characterize recent acquisition history or recent acquisition history associated with a particular client; inputting an input data set including communication statistics into a trained machine learning model; the communication statistics include: counts of multiple acquisitions or multiple outgoing acquisitions; and duration statistics of multiple acquisitions; length statistics of multiple outgoing acquisitions; subset diversity statistics related to a plurality of different data collections in communication therewith; a plurality of the data sets comprising data characterizing each of the data sensors of the particular client, processing a plurality of the data sets comprising: based on the data sensor measurements, generating transmission statistics representing the duration, intensity, or frequency of movement of a particular client; inputting an input dataset including communication statistics into a trained machine learning model, determining based on the results that alert conditions are met; and The result of the alarm condition, the result is sent to another client; so that the user can collect according to the alarm or the process of collecting or transmitting the data, in addition, in the embodiment, by alarming specific data or transmission rules, Effectively ensure that the data can be alarmed when the data is collected or converted, so that the data can be effectively supervised during the transmission process, and the security of the entire transmission process can be improved;

所述使用方法还包括：所述客户端基于多个用户的数据来访问用户分类模型；以及其中确定要在特定客户端处监视数据以预测事件的步骤包括：由另一客户端使用用户分类模型将用户分类为用户组；另一客户端至少部分地基于对用户进行分类的用户组来选择训练有素的机器学习模型，由另一所述客户端在多个所述用户的数据的至少一部分上执行功能以产生组表达；以及所述另一客户端将所述组表达式映射到N维空间；所述另一客户端将所述组表达式所包围的区域分类为用户组；具体的，在本实施例中，通过所述客户端对多个用户的数据来对所述用户类型的分类，使得对用户进行针对性的实时的数据的传输或者推送的操作；在本实施例中，还通过由另一客户端使用用户分类模型将用户分类为用户组；另一客户端至少部分地基于对用户进行分类的用户组来选择训练有素的机器学习模型，由另一所述客户端在多个所述用户的数据的至少一部分上执行功能以产生组表达，保证各个所述客户端在进行数据传输或者采集的过程中，能够对各个客户端进行分配保证在进行采集的过程中，不会存在数据进行相互的干扰，有效的提高与所述采集器进行连接的各个所述客户端的数据交互的高效，不会存在占用或者相互之间的干扰；另外，各个所述客户端在进行数据传输的过程中，对所述数据侧标识通过产生的组表达进行标记，防止各个所述数据在使用的过程中，对对应的数据产生干扰或者影响；The method of use further includes: the client accessing a user classification model based on data for a plurality of users; and wherein the step of determining that data is to be monitored at a particular client to predict events includes using, by another client, the user classification model classifying users into user groups; another client selects a trained machine learning model based at least in part on the user groups classifying the users, and another said client selects a trained machine learning model based on at least a portion of the data for a plurality of said users and the another client maps the group expression to an N-dimensional space; the another client classifies the area enclosed by the group expression as a user group; the specific , in this embodiment, the user type is classified by the client to the data of multiple users, so that targeted real-time data transmission or push operation is performed on the user; in this embodiment, The user is also classified into groups of users by using the user classification model by another client; Perform a function on at least a part of the data of the plurality of users to generate a group representation, to ensure that each of the clients can be allocated to each client during the process of data transmission or collection to ensure that during the process of collection, There will be no mutual interference of data, effectively improving the efficiency of data interaction of each of the clients connected to the collector, and there will be no occupation or mutual interference; In the process of data transmission, marking the group expression generated by the data side identification to prevent each of the data from interfering or affecting the corresponding data in the process of use;

实施例三：本实施例应当理解为至少包含前述任一一个实施例的全部特征，并在其基础上进一步改进，具体的，提供一种大数据信息采集***，所述采集装置包括采集端口4、输出端口2、显示装置1和若干个数据传感器连接装置，所述采集装置与特定客户端配对联合使用，保证所述采集装置对数据的采集的过程中，能够根据实际的需要对各个所述数据连接通道的传输或者连接，具体的，所述采集端口4和所述输出端口2分别设置在采集装置本体5的两侧，且所述采集装置还包括控制面板3和数据分发装置，所述控制面板3设置与所述显示装置1同一侧，使得所述显示装置1能够针对所述显示装置1显示的画面或者显示的参数进行调整的操作，在本实施例中，所述采集端口4被构造为与各个所述特定客户端或者移动电子设备进行数据链路的连接，使得传输的所述数据或者采集的数据能够通过所述采集装置的采集的操作，进行实时的控制；所述数据分发装置被构造为对所述数据的传输之间进行数据的分发的操作，保证所述数据在进行采集或者分发的过程中能够根据实际的需要进行操作，有效保证所述数据采集或者传输的高效的进行；另外，在本实施例中，所述输出端口2还被配置为对所述存储器或者云服务器或者所述采集网络之间相互连接，行程数据的传输或者采集的通道；在本实施例中，还提供一种大数据信息采集***的使用方法，所述使用方法包括：由节点集群的拥有节点确定并包括处理器的第一数据集，该第一数据集表示与该节点集群的拥有节点拥有的数据结构相对应的拥有块；由节点集群的拥有节点确定第二数据集，该第二数据集表示节点集群中正在使用的拥有块中的已使用拥有块；基于所述第一数据集与所述第二数据集的差，由所述节点簇的所属节点确定代表所述节点簇中未使用的所述拥有块的未使用的拥有块的第三数据集；然后基于第三数据集，由节点集群的拥有节点进行数据收集，以收集节点集群中未使用的未使用的拥有的块；确定表示所述节点集群中正在使用的所拥有块的第二数据集包括：确定与拥有节点正在使用的所使用的所拥有块相对应的第一组零个或多个标识符，获得第二组零个或多个标识符，它们对应于节点集群的一个或多个非拥有节点正在使用的其他已使用的拥有的块，并将第一组和第二组组合到第二数据集中；所述数据结构包括树，并且其中，确定与所述拥有节点正在使用的所使用的所拥有的块相对应的第一组零个或多个标识符包括遍历所述所有者的拥有节点所拥有的树；深度优先遍历的节点簇以定位零个或多个与正在使用的已使用拥有块相对应的块标识符；由所述节点集群的拥有节点维护代表使用中的所使用的所拥有的区块的最近添加的区块标识符的高速缓存，其中，确定第一组零或多个标识符对应于拥有节点正在使用的已使用拥有块包括访问高速缓存以消除遍历树期间位于使用中的重复块；将所述第一组和所述第二组组合到所述第二数据集中包括去除重复的标识符；获得与所述节点集群的一个或多个非拥有节点正在使用的所述其他使用过的拥有的大块相对应的第二组零个或多个标识符包括从所述一个或多个中的每个接收节点集群的更多非拥有节点，相应的数据结构包含零个或多个标识符，这些标识符对应于一个或多个非拥有节点正在使用的其他已使用拥有块；由所述节点集群的拥有节点持久存储来自所述节点集群中的所述一个或多个非拥有节点中的每一个的相应数据结构；由所述节点集群的所述拥有节点执行所述节点集群中的参考计数，以根据定义的充足性准则，通过数据传输来确定是否有足够数量的块潜在地可传输，并且响应确定足够数量的数据块被确定为可能可传输，并基于第三数据集，由节点群集的拥有节点调度数据传输操作，该操作包括数据传输未使用拥有的数据块节点集群；由所述节点集群的所述拥有节点确定第四数据集，所述第四数据集表示由所述拥有节点正在使用并且由所述节点集群的除了所述节点之外的其他节点所拥有的非拥有块。Embodiment 3: This embodiment should be understood to include at least all the features of any one of the foregoing embodiments, and further improve on the basis thereof. Specifically, a big data information collection system is provided, and the collection device includes a collection port. 4. Output port 2, display device 1 and several data sensor connection devices, the acquisition device is paired with a specific client and used in conjunction to ensure that in the process of data acquisition by the acquisition device, it can be used according to actual needs. The transmission or connection of the data connection channel, specifically, the collection port 4 and the output port 2 are respectively arranged on both sides of the collection device body 5, and the collection device also includes a control panel 3 and a data distribution device, so The control panel 3 is set on the same side as the display device 1, so that the display device 1 can adjust the screen or parameters displayed by the display device 1. In this embodiment, the collection port 4 It is configured to perform data link connection with each of the specific clients or mobile electronic devices, so that the transmitted data or the collected data can be controlled in real time through the collection operation of the collection device; the data The distribution device is configured to perform the operation of data distribution between the data transmissions, to ensure that the data can be operated according to actual needs during the process of collection or distribution, and to effectively ensure the efficiency of the data collection or transmission. In addition, in this embodiment, the output port 2 is also configured as a channel for interconnecting the storage or cloud server or the collection network, transmission or collection of travel data; in this embodiment , a method for using a big data information collection system is also provided, the using method includes: a first data set determined by an owning node of a node cluster and including a processor, the first data set representing ownership of the node cluster an owning block corresponding to a data structure owned by a node; a second data set is determined by the owning node of the node cluster, the second data set representing a used owned block among the owned blocks being used in the node cluster; based on the first data The difference between the set and the second data set, the third data set representing the unused owned blocks of the owned blocks that are not used in the node cluster is determined by the node to which the node cluster belongs; and then based on the third data set set, data collection is performed by the owning node of the node cluster to collect unused and unused owned blocks in the node cluster; determining the second data set representing the owned blocks being used in the node cluster includes: determining and owning A first set of zero or more identifiers corresponding to the used owned blocks being used by the node, and a second set of zero or more identifiers corresponding to one or more non-owning nodes of the node cluster being using other used owned blocks and combining the first and second sets into a second data set; the data structure includes a tree, and wherein the used owned blocks that are in use with the owning node are determined The first set of zero or more identifiers corresponding to the block consists of traversing the tree owned by the owner's owning node; the depth-first traversal of the node cluster is zero or more block identifiers corresponding to used owned blocks in use; the most recently added block identifier representing the used owned block in use is maintained by the owning node of the node cluster , wherein determining that the first set of zero or more identifiers corresponds to a used owned block in use by the owning node includes accessing the cache to eliminate duplicate blocks in use during tree traversal; combining the first set and The second set of combining into the second dataset includes deduplicating identifiers; obtaining an identifier corresponding to the other used owned chunks being used by one or more non-owning nodes of the cluster of nodes; A second set of zero or more identifiers includes more non-owning nodes from each of the one or more receiving node clusters, the corresponding data structures containing zero or more identifiers corresponding to other used owning blocks in use by one or more non-owning nodes; the corresponding data structures from each of the one or more non-owning nodes in the node cluster are persistently stored by the owning node of the node cluster ; performing reference counting in the node cluster by the owning node of the node cluster to determine, through data transfer, whether a sufficient number of blocks are potentially transferable according to defined adequacy criteria, and responsive to determining a sufficient number The data blocks are determined to be potentially transferable, and based on the third data set, a data transfer operation is scheduled by the owning node of the node cluster, the operation including the data transfer not using the owned data block node cluster; The owning node determines a fourth data set representing non-owning blocks in use by the owning node and owned by other nodes of the node cluster other than the node.

在上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述或记载的部分，可以参见其它实施例的相关描述。In the foregoing embodiments, the description of each embodiment has its own emphasis. For parts that are not described or described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

综上所述，本发明的一种大数据信息采集***及使用方法，通过将输入数据分为针对实时数据流的第一部分和针对非实时数据流的第二部分表示在网络上执行的多路径处理的形式，保证数据在传输或者采集的过程中，能够进行大量数据采集和存储，提高整个***对大量数据的高效的采集；通过确定输入数据的量超过阈值，阀值包括数据最多的阀值量，则这触发基于云的功能向存储器的迁移以及对基于云的功能的选择，输入数据的某些部分，使用迁移的功能在存储器进行处理；通过在网络高延迟的情况下，使用迁移功能在存储器处理的输入数据可能是需要实时分析的数据，在采集网络中，执行迁移的功能的存储器设备可能比云计算机更靠近数据源，从而消除了如果将数据直接发送给高延迟会导致的处理延迟，由从采集网络迁移到存储器的功能执行的处理包括数据清理、数据过滤、数据标准化、数据转换、数据汇总、数据分析或任何其他合适形式的数据处理；通过将包括通信统计信息的输入数据集输入到训练后的机器学习模型，并基于结果确定满足警报条件，使得使用者能够根据报警或者数据的采集或者传输的过程进行采集，对特定数据或者传输规则进行报警，有效保证数据进行采集或者转换等功能时能够报警，实现对数据在传输的过程中能够有效的进行监管，提高整个传输过程的安全性；通过采用客户端在多个用户的数据的至少一部分上执行功能以产生组表达，保证各个客户端在进行数据传输或者采集的过程中，能够对各个客户端进行分配保证在进行采集的过程中，不会存在数据进行相互的干扰，有效的提高与采集器进行连接的各个客户端的数据交互的高效，不会存在占用或者相互之间的干扰。In summary, a big data information collection system and method of use of the present invention, by dividing the input data into a first part for real-time data flow and a second part for non-real-time data flow, represent multi-path execution on the network The form of processing ensures that a large amount of data can be collected and stored in the process of data transmission or collection, and improves the efficient collection of large amounts of data by the entire system; by determining that the amount of input data exceeds the threshold, the threshold includes the threshold with the most data volume, then this triggers the migration of cloud-based functions to storage and the selection of cloud-based functions, some parts of the input data, are processed in storage using the migrated function; by using the migrated function in the case of high network latency The input data processed in memory may be data that needs to be analyzed in real time, and in the acquisition network, the memory device that performs the function of the migration may be closer to the data source than the cloud computer, eliminating the processing that would result if the data was sent directly to high latency Delay, processing performed by functions migrating from acquisition network to storage including data cleaning, data filtering, data normalization, data transformation, data aggregation, data analysis or any other suitable form of data processing; by incorporating incoming data including communication statistics Set input to the trained machine learning model, and determine the alarm conditions based on the results, so that users can collect according to the process of alarm or data collection or transmission, alarm specific data or transmission rules, and effectively ensure data collection or transmission. It can alarm when converting and other functions, realize the effective supervision of data in the process of transmission, and improve the security of the entire transmission process; by using the client to perform functions on at least part of the data of multiple users to generate group expressions, Ensure that each client can allocate each client in the process of data transmission or collection to ensure that there is no mutual interference of data during the collection process, effectively improving the quality of each client connected to the collector. Data interaction is efficient, and there will be no occupancy or mutual interference.

虽然上面已经参考各种实施例描述了本发明，但是应当理解，在不脱离本发明的范围的情况下，可以进行许多改变和修改。也就是说上面讨论的方法，***和设备是示例。各种配置可以适当地省略，替换或添加各种过程或组件。例如，在替代配置中，可以以与所描述的顺序不同的顺序执行方法，和/或可以添加，省略和/或组合各种部件。而且，关于某些配置描述的特征可以以各种其他配置组合，如可以以类似的方式组合配置的不同方面和元素。此外，随着技术发展其中的元素可以更新，即许多元素是示例，并不限制本公开或权利要求的范围。While the invention has been described above with reference to various embodiments, it should be understood that many changes and modifications can be made without departing from the scope of the invention. That said, the methods, systems and apparatus discussed above are examples. Various configurations may omit, substitute or add various procedures or components as appropriate. For example, in alternative configurations, the methods may be performed in an order different from that described, and/or various components may be added, omitted, and/or combined. Furthermore, features described with respect to certain configurations may be combined in various other configurations, eg, different aspects and elements of the configurations may be combined in a similar manner. Furthermore, elements therein may be updated as technology develops, ie, many of the elements are examples and do not limit the scope of the disclosure or the claims.

在说明书中给出了具体细节以提供对包括实现的示例性配置的透彻理解。然而，可以在没有这些具体细节的情况下实践配置例如，已经示出了众所周知的电路，过程，算法，结构和技术而没有不必要的细节，以避免模糊配置。该描述仅提供示例配置，并且不限制权利要求的范围，适用性或配置。相反，前面对配置的描述将为本领域技术人员提供用于实现所描述的技术的使能描述。在不脱离本公开的精神或范围的情况下，可以对元件的功能和布置进行各种改变。Specific details are given in the description to provide a thorough understanding of example configurations, including implementations. However, configurations may be practiced without these specific details. For example, well-known circuits, procedures, algorithms, structures and techniques have been shown without unnecessary detail in order to avoid obscuring the configurations. This description provides example configurations only, and does not limit the scope, applicability, or configuration of the claims. Rather, the foregoing descriptions of configurations will provide those skilled in the art with an enabling description for implementing the described techniques. Various changes may be made in the function and arrangement of elements without departing from the spirit or scope of the disclosure.

综上，其旨在上述详细描述被认为是例示性的而非限制性的，并且应当理解，以上这些实施例应理解为仅用于说明本发明而不用于限制本发明的保护范围。在阅读了本发明的记载的内容之后，技术人员可以对本发明作各种改动或修改，这些等效变化和修饰同样落入本发明权利要求所限定的范围。In conclusion, it is intended that the above detailed description is to be considered as illustrative rather than restrictive, and it should be understood that these embodiments above should be understood to be merely illustrative of the present invention and not intended to limit the scope of protection of the present invention. After reading the contents of the description of the present invention, the skilled person can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

一种大数据信息采集***，其特征在于，包括处理器、客户端、采集器和各个数据传感器，所述处理器配置为执行以下操作的指令：确定要在特定客户端上监视数据以指示事件；并响应于该确定：接收代表在特定客户端处收集的多个输入或数据传感器测量值的多个数据集；使用训练有素的机器学习模型处理多个数据集以生成与事件的预测风险相对应的结果，并在显示设备上输出结果。A big data information collection system, characterized in that it includes a processor, a client, a collector and various data sensors, the processor is configured to execute instructions for the following operations: determining that data is to be monitored on a specific client to indicate an event ; and in response to the determination: receive multiple datasets representing multiple inputs or data sensor measurements collected at a particular client; process the multiple datasets using a trained machine learning model to generate a predicted risk associated with the event Corresponding results, and output the results on the display device.

2.如权利要求1所述的一种大数据信息采集***及使用方法，其特征在于，所述采集器和各个所述数据采集传感器之间对所述数据进行采集的操作，保证所述数据在被采集的过程中，能够对大量所述数据进行收集并就通过所述客户端进行分发的操作。2 . The big data information collection system and method of use according to claim 1 , wherein the operation of collecting the data between the collector and each of the data collection sensors ensures that the data is collected. 3 . In the process of being collected, a large amount of the data can be collected and distributed through the client.

3.如权利要求1所述的一种大数据信息采集***及使用方法，其特征在于，所述多个数据集包括标识特定所述客户端所位于的多个位置的位置数据，处理多个所述数据集包括：标识多个基础数据；特定客户端经常位于的位置区域；使用位置数据确定指示该特定客户端在多个基本位置区域之外的时间变量；并将包括时间变量的输入数据集输入到训练后的机器学习模型，处理多个所述数据集包括：使用所述多个数据集，确定表征与所述特定客户端相关联的最近采集历史或最近采集历史的通信统计；将包括通信统计信息的输入数据集输入到训练后的机器学习模型。3 . The big data information collection system and method of use according to claim 1 , wherein the multiple data sets include location data identifying multiple locations where the specific client is located, and processing multiple data sets. 4 . The data set includes: identifying a plurality of base data; location areas in which a particular client is frequently located; using the location data to determine a time variable indicating that the particular client is outside the plurality of base location areas; and including input data for the time variable set input to the trained machine learning model, and processing a plurality of the data sets includes: using the plurality of data sets, determining communication statistics that characterize the most recent acquisition history or recent acquisition history associated with the particular client; An input dataset including communication statistics is fed into the trained machine learning model.

4.如权利要求1所述的一种大数据信息采集***，其特征在于，所述采集器包括网络设备、云计算机、存储器和可执行指令，所述可执行指令存储在所述存储器中，所述处理器被配置为访问至少一个所述存储器并执行所述计算机可执行指令以：接收输入数据；以及至少部分地基于多个分类标准对输入数据进行分类以获得分类的输入数据；至少部分地基于以下至少一项来确定基于云的数据处理功能的第一部分，以从多个云计算机传输到网络设备中：分类的输入数据；多个网络特征；采集网络或网络路径中的多个设备特征；将基于云的数据处理功能的第一部分从多个云计算机传输到网络设备，以在网络设备获得传输的数据处理功能；在第一网络设备上，使用采集的数据处理功能处理输入数据的至少一个子集。4. The big data information collection system according to claim 1, wherein the collector comprises a network device, a cloud computer, a memory and an executable instruction, and the executable instruction is stored in the memory, The processor is configured to access at least one of the memories and execute the computer-executable instructions to: receive input data; and classify the input data based at least in part on a plurality of classification criteria to obtain classified input data; at least in part determining a first portion of a cloud-based data processing function for transmission from a plurality of cloud computers into a network device based on at least one of: classified input data; a plurality of network characteristics; collecting a network or a plurality of devices in a network path Features; transmitting the first part of the cloud-based data processing function from a plurality of cloud computers to a network device to obtain the transmitted data processing function at the network device; on the first network device, using the collected data processing function to process the input data. at least a subset.

5.如权利要求1所述的一种大数据信息采集***，其特征在于，所述处理器被配置为通过执行所述计算机可执行指令以确定所述输入数据的量超过阈值来对所述输入数据进行分类，所述处理器被配置为执行计算机可执行指令以：至少部分地响应于确定输入数据的数量超过阈值，使用传输的数据处理功能选择输入数据的一个子集以进行处理；使用基于云的数据处理功能的第二部分，将处理后的输入数据的一个子集发送到多个云计算机中以进行其他数据处理。5. A big data information collection system according to claim 1, wherein the processor is configured to perform the processing on the data by executing the computer-executable instructions to determine that the amount of the input data exceeds a threshold. classifying the input data, the processor configured to execute computer-executable instructions to: select a subset of the input data for processing using the transmitted data processing function in response, at least in part, to determining that the amount of the input data exceeds a threshold; using The second part of the cloud-based data processing function sends a subset of the processed input data to multiple cloud computers for additional data processing.

6.如权利要求1所述的一种大数据信息采集***，其特征在于，所述处理器还被配置为执行所述计算机可执行指令以：确定网络等待时间超过阈值等待时间；以及部分地响应于确定网络等待时间超过阈值等待时间，使用传输的数据处理功能选择输入数据的一个子集以进行处理；使用基于云的数据处理功能的第二部分，将处理后的输入数据的至少一个子集发送到多个云计算机中以进行数据的处理。6. The big data information collection system of claim 1, wherein the processor is further configured to execute the computer-executable instructions to: determine that a network latency exceeds a threshold latency; and partially In response to determining that the network latency exceeds the threshold latency, a subset of the input data is selected for processing using the transmitted data processing function; at least a subset of the processed input data is processed using a second portion of the cloud-based data processing function The set is sent to multiple cloud computers for data processing.

7.如权利要求6所述的一种大数据信息采集***，其特征在于，确定网络等待时间的网络特征影响功能是否从采集网络迁移到存储器；如果网络等待时间超过阈值等待时间，则将基于云的功能的某些部分迁移到存储器，以允许至少一部分输入数据绕过高网络等待时间并使用迁移的功能在存储器进行处理；由从采集网络迁移到存储器的功能执行的处理包括数据清理、数据过滤、数据标准化、数据转换、数据汇总、数据分析或任何其他合适形式的数据处理；此外，从采集网络迁移到存储器的功能的类型是上下文或应用特定的；提供实时数据的短期分析的功能迁移到存储器，而提供更多长期分析的其他功能则保留在采集网络中；基于云的数据处理功能包括数据分析数据过滤对过滤后的数据转换；使用云数据库内的另一数据源对转换后的数据进行分析，出于审计目的而存储原始数据，并存储分析结果以立即使用；如果要考虑将大量数据传输到云或由于网络带宽而导致网络延迟，则将一些基于云的功能迁移到沿任何地方的一个或多个设备网络路径。7. A kind of big data information collection system as claimed in claim 6, it is characterized in that, determine whether the network characteristic influence function of network waiting time migrates from collecting network to memory; If network waiting time exceeds threshold waiting time, will be based on Portions of functions of the cloud are migrated to storage to allow at least a portion of the input data to bypass high network latency and be processed in storage using the migrated functions; processing performed by functions migrated from the acquisition network to storage includes data cleaning, data Filtering, data normalization, data transformation, data summarization, data analysis, or any other suitable form of data processing; in addition, the type of functionality migrated from the acquisition network to storage is context or application specific; functionality migration that provides short-term analysis of real-time data to storage, while other functions that provide more long-term analysis remain in the acquisition network; cloud-based data processing functions include data analysis, data filtering, and transformation of filtered data; Data is analyzed, raw data is stored for auditing purposes, and analysis results are stored for immediate use; if large amounts of data are to be transferred to the cloud or network latency due to network bandwidth is to be considered, migrate some cloud-based functionality to any One or more device network paths in place.

8.一种大数据信息使用方法，其特征在于，所述使用方法包括：使用多个所述数据集，确定表征与特定客户端相关联的最近采集历史或最近采集历史的通信统计；将包括通信统计信息的输入数据集输入到训练后的机器学习模型；所述通信统计量包括：多个采集或多个外发采集的计数；以及多个采集的持续时间统计；多个外发采集的长度统计；与多个采集或多个传出采集正在与之通信的多个不同数据采集有关的子集多样性统计；多个所述数据集包括表征所述特定客户端的各个所述数据传感器的数据，处理多个所述数据集包括：基于所述数据传感器测量值，生成传输统计量代表特定客户端的持续时间、强度或移动频率；将包括通信统计信息的输入数据集输入到训练后的机器学习模型，基于结果确定满足警报条件；以及作为确定满足警报条件的结果，将结果发送到另一客户端。8. A method of using big data information, characterized in that, the method of using comprises: using a plurality of the data sets to determine communication statistics representing recent collection histories or recent collection histories associated with a particular client; including An input dataset of communication statistics is input to the trained machine learning model; the communication statistics include: counts of multiple acquisitions or multiple outgoing acquisitions; and duration statistics of multiple acquisitions; Length statistics; subset diversity statistics related to multiple acquisitions or multiple different data acquisitions with which multiple outgoing acquisitions are communicating; a plurality of said data sets including data for each of said data sensors that characterize said particular client data, processing a plurality of said data sets comprises: based on said data sensor measurements, generating transmission statistics representing the duration, intensity or frequency of movement of a particular client; inputting an input data set including communication statistics into the trained machine Learning the model, determining that the alert condition is satisfied based on the result; and sending the result to another client as a result of determining that the alert condition is satisfied.

9.如权利要求8所述的一种大数据信息使用方法，其特征在于，所述使用方法还包括：所述客户端基于多个用户的数据来访问用户分类模型；以及其中确定要在特定客户端处监视数据以预测事件的步骤包括：由另一客户端使用用户分类模型将用户分类为用户组；另一客户端至少部分地基于对用户进行分类的用户组来选择训练有素的机器学习模型，由另一所述客户端在多个所述用户的数据的至少一部分上执行功能以产生组表达；以及所述另一客户端将所述组表达式映射到N维空间；所述另一客户端将所述组表达式所包围的区域分类为用户组。9. The method for using big data information according to claim 8, wherein the method further comprises: the client accesses a user classification model based on data of multiple users; The step of monitoring data at the client to predict events includes: classifying, by another client, users into groups of users using a user classification model; and selecting, by another client, a trained machine based at least in part on the groups of users classifying the users a learning model by another of said clients performing a function on at least a portion of a plurality of said users' data to generate group expressions; and said another client mapping said group expressions to an N-dimensional space; said Another client classifies the area enclosed by the group expression as a user group.

10.如权利要求9所述的一种大数据信息使用方法，其特征在于，通过所述客户端对多个用户的数据来对所述用户类型的分类，使得对用户进行针对性的实时的数据的传输或者推送的操作；还通过由另一客户端使用用户分类模型将用户分类为用户组；另一客户端至少部分地基于对用户进行分类的用户组来选择训练有素的机器学习模型，由另一所述客户端在多个所述用户的数据的至少一部分上执行功能以产生组表达；另外，各个所述客户端在进行数据传输的过程中，对所述数据侧标识通过产生的组表达进行标记。10 . The method for using big data information according to claim 9 , wherein the user types are classified by the client to the data of multiple users, so that targeted real-time information is performed on the users. 11 . an operation of transferring or pushing data; also by classifying users into groups of users by using a user classification model by another client; another client selects a trained machine learning model based at least in part on the groups of users that classify users , another said client performs a function on at least a part of the data of a plurality of said users to generate a group expression; in addition, in the process of data transmission, each said client side identifies the data side by generating The group expression was marked.