WO2020015067A1

WO2020015067A1 - Data acquisition method, device, equipment and storage medium

Info

Publication number: WO2020015067A1
Application number: PCT/CN2018/103791
Authority: WO
Inventors: 白红丽
Original assignee: 平安科技（深圳）有限公司
Priority date: 2018-07-19
Filing date: 2018-09-03
Publication date: 2020-01-23
Also published as: CN110019486A; CN110019486B

Abstract

Disclosed are a data acquisition method, device, equipment and a computer-readable storage medium. The method comprises: acquiring a target field set in a service requirement when the service requirement is received, and generating an acquisition target table according to the target field set (S10); matching the target field set with a preset acquisition library, and judging whether a target acquisition rule matched with the target field set exists in the preset acquisition library, wherein the target acquisition rule comprises a related table identifier and a data acquisition path (S20); if the target acquisition rule exists, determining a corresponding target form according to the related table identifier, and associating the tables in the target form (S30); acquiring target data corresponding to the target field set in an association table of the target form according to the data acquisition path, and storing the target data to the acquisition target table (S40). By means of the method, automatic acquisition of target data is realized, manpower is saved and acquisition efficiency is improved.

Description

数据采集方法、装置、设备及存储介质 Data acquisition method, device, equipment and storage medium Ranch

本申请要求于2018年7月19日提交中国专利局、申请号为201810794561.1、发明名称为“数据采集方法、装置、设备及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在申请中。This application claims priority from a Chinese patent application filed on July 19, 2018 with the Chinese Patent Office, application number 201810794561.1, and the invention name is "data acquisition method, device, device and storage medium", the entire contents of which are incorporated by reference Applying.

技术领域Technical field

本申请涉及数据处理领域，尤其涉及一种数据采集方法、装置、设备及计算机可读存储介质。The present application relates to the field of data processing, and in particular, to a data collection method, device, device, and computer-readable storage medium.

背景技术Background technique

随着大数据时代的来临，数据库中的数据量变得越来越大，因此，数据的查询耗时也越来越长。而在日常工作中，业务用户经常需要大量的数据做统计分析，但是不同的用户需求有差异且需求变化太快，无法通过报表来进行支持，因此，需要通过数据采集进行数据提取，如通过某种程序或脚本以某一规则完成数据库的数据查询与获取。由于各个数据采集需求不同，因此需要针对每个采集需求进行分析然后编写对应脚本。但是现有数据采集需要人工进行需求分析与脚本编写操作，不仅浪费了大量人力，而且需要重复进行分析与编写操作，导致采集效率低下。因此，如何解决现有采集数据技术浪费人力以及效率低下的技术问题，成为了目前亟待解决的技术问题。 With the advent of the era of big data, the amount of data in the database has become larger and larger, so the query of data has also taken longer and longer. In daily work, business users often need a large amount of data for statistical analysis. However, different users have different needs and the requirements change too quickly to be supported by reports. Therefore, data collection is required for data extraction, such as through a certain This kind of program or script completes the data query and acquisition of the database with a certain rule. Because each data collection requirement is different, you need to analyze each collection requirement and write a corresponding script. However, the existing data collection requires manual demand analysis and script writing operations, which not only wastes a lot of manpower, but also requires repeated analysis and writing operations, resulting in low collection efficiency. Therefore, how to solve the technical problems of wasting manpower and inefficiency of the existing data collection technology has become a technical problem to be solved urgently.

发明内容Summary of the invention

本申请的主要目的在于提供一种数据采集方法、装置、设备及计算机可读存储介质，旨在解决现有采集数据技术浪费人力以及效率低下的技术问题。The main purpose of this application is to provide a data collection method, device, device, and computer-readable storage medium, which are aimed at solving the technical problems of wasting manpower and low efficiency of existing data collection technologies.

为实现上述目的，本申请提供一种数据采集方法，所述数据采集方法包括以下步骤：To achieve the above object, the present application provides a data collection method, and the data collection method includes the following steps:

在接收到业务需求时，获取所述业务需求中的目标字段集合，并根据所述目标字段集合生成采集目的表；When receiving a business demand, obtaining a target field set in the business demand, and generating a collection destination table according to the target field set;

将所述目标字段集合与预设采集库进行匹配，判断所述预设采集库中是否存在与所述目标字段集合匹配的目标采集规则，其中，所述目标采集规则包括相关表格标识以及数据采集路径；Matching the target field set with a preset collection library to determine whether there is a target collection rule matching the target field set in the preset collection library, wherein the target collection rule includes a related form identifier and data collection path;

若判定所述预设采集库中存在所述目标采集规则，则根据所述相关表格标识，确定对应的目标表单，将所述目标表单中的表格进行关联；If it is determined that the target collection rule exists in the preset collection library, determining a corresponding target form according to the related form identifier, and correlating the forms in the target form;

根据所述数据采集路径，在所述目标表单的关联表格中获取所述目标字段集合对应的目标数据，并将所述目标数据存储至所述采集目的表。According to the data collection path, obtain target data corresponding to the target field set in an association table of the target form, and store the target data in the collection destination table.

可选地，所述将所述目标字段集合与预设采集库进行匹配，判断所述预设采集库中是否存在与所述目标字段集合匹配的目标采集规则，其中，所述目标采集规则包括对应关联关系以及数据采集路径的步骤包括：Optionally, the target field set is matched with a preset collection library to determine whether a target collection rule matching the target field set exists in the preset collection library, wherein the target collection rule includes The steps of corresponding association relationship and data collection path include:

将所述目标字段集合中的各个目标字段与所述预设采集库中的预设字段集合进行匹配，并计算所述目标字段集合与所述预设字段集合的匹配度；Matching each target field in the target field set with a preset field set in the preset collection library, and calculating a degree of matching between the target field set and the preset field set;

根据所述匹配度，判断所述预设采集库中是否存在与所述目标字段集合匹配的目标采集规则，其中，与所述目标字段集合的匹配度不小于预设阈值的预设字段集合对应的采集规则为所述目标采集规则。Determining, according to the matching degree, whether a target collection rule matching the target field set exists in the preset collection library, wherein the preset field set corresponding to the target field set is not less than a preset threshold value The collection rule is the target collection rule.

可选地，所述根据所述匹配度，判断所述预设采集库中是否存在与所述目标字段集合匹配的目标采集规则，其中，与所述目标字段集合的匹配度不小于预设阈值的预设字段集合对应的采集规则为所述目标采集规则的步骤之后，还包括：Optionally, according to the matching degree, determining whether a target collection rule matching the target field set exists in the preset collection library, and the matching degree with the target field set is not less than a preset threshold After the collection rule corresponding to the preset field set is the target collection rule, the method further includes:

若判定所述预设采集库中不存在所述目标采集规则，则获取与所述目标字段集合的匹配度最大的预设字段集合，作为相关字段集合；If it is determined that the target collection rule does not exist in the preset collection library, obtaining a preset field set that has the greatest degree of matching with the target field set as a related field set;

获取所述相关字段集合对应的采集规则，作为相关采集规则，并将所述相关采集规则推送至当前终端，以便用户参考。Acquire a collection rule corresponding to the related field set as a related collection rule, and push the related collection rule to a current terminal for reference by a user.

可选地，所述若判定所述预设采集库中存在所述目标采集规则，则根据所述相关表格标识，确定对应的目标表单，将所述目标表单中的表格进行关联的步骤包括：Optionally, if it is determined that the target collection rule exists in the preset collection library, determining a corresponding target form according to the related form identifier, and correlating the forms in the target form include:

若所述预设采集库中存在所述目标采集规则，则获取所述目标采集规则的规则数目，并判断所述规则数目是否唯一；If the target collection rule exists in the preset collection database, obtaining the number of rules of the target collection rule, and determining whether the number of rules is unique;

若所述规则数目唯一，则根据所述目标采集规则中的相关表格标识，确定对应的目标表单，将所述目标表单中的表格进行关联。If the number of rules is unique, a corresponding target form is determined according to a related form identifier in the target collection rule, and the forms in the target form are associated.

可选地，所述若所述预设采集库中存在所述目标采集规则，则获取所述目标采集规则的规则数目，并判断所述规则数目是否唯一的步骤之后，还包括：Optionally, after the step of obtaining the target collection rule number and determining whether the number of rules is unique if the target collection rule exists in the preset collection database, the method further includes:

若所述规则数目不唯一，则获取所述目标采集规则中的待定采集规则以及对应的匹配度；If the number of rules is not unique, obtaining a pending collection rule and a corresponding matching degree among the target collection rules;

根据所述匹配度大小，将所述待定采集规则进行排序，生成并显示对应的待定采集规则列表；Sorting the pending collection rules according to the matching degree to generate and display a corresponding list of pending collection rules;

基于所述待定采集规则列表，接收用户选择操作触发的选择指令，并根据所述选择指令，在所述待定采集规则列表中确定目标采集规则。 Based on the pending collection rule list, a selection instruction triggered by a user selection operation is received, and a target collection rule is determined in the pending collection rule list according to the selection instruction.

若所述规则数目不唯一，则获取所述目标采集规则中的各个采集规则，作为待选采集规则；If the number of rules is not unique, obtaining each collection rule in the target collection rule as a candidate collection rule;

根据所述待选采集规则中表格标识，确定所述各个待选采集规则对应的表格数量，并获取表格数量最小的待选采集规则，作为目标采集规则。According to the table identifier in the candidate collection rule, the number of tables corresponding to each candidate collection rule is determined, and the candidate collection rule with the smallest number of tables is obtained as the target collection rule.

可选地，所述将所述目标字段集合与预设采集库进行匹配，判断所述预设采集库中是否存在与所述目标字段集合匹配的目标采集规则，其中，所述目标采集规则包括相关表格标识以及数据采集路径的步骤之后，还包括：Optionally, the target field set is matched with a preset collection library to determine whether a target collection rule matching the target field set exists in the preset collection library, wherein the target collection rule includes After the steps of identifying the relevant form and data collection path, it also includes:

若判定所述预设采集库中不存在所述目标采集规则，则将所述目标字段集合上报至预设处理端；If it is determined that the target collection rule does not exist in the preset collection library, report the target field set to a preset processing end;

在接收到所述预设处理端反馈的采集规则时，将所述预设处理端反馈的采集规则作为目标采集规则，并将所述目标字段集合和所述目标采集规则对应存储至所述预设采集库，以更新所述预设采集库。When receiving the collection rule feedback from the preset processing end, the collection rule feedback from the preset processing end is used as the target collection rule, and the target field set and the target collection rule are correspondingly stored in the preset A collection library is set to update the preset collection library.

此外，为实现上述目的，本申请还提供一种数据采集装置，所述数据采集装置包括： In addition, in order to achieve the foregoing objective, the present application further provides a data acquisition device, where the data acquisition device includes:

表格生成模块，用于在接收到业务需求时，获取所述业务需求中的目标字段集合，并根据所述目标字段集合生成采集目的表；A form generating module, configured to obtain a target field set in the business demand when a business demand is received, and generate a collection destination table according to the target field set;

规则判断模块，用于将所述目标字段集合与预设采集库进行匹配，判断所述预设采集库中是否存在与所述目标字段集合匹配的目标采集规则，其中，所述目标采集规则包括相关表格标识以及数据采集路径；A rule judgment module is configured to match the target field set with a preset collection library, and determine whether a target collection rule matching the target field set exists in the preset collection library, where the target collection rule includes Relevant form identification and data collection path;

表格关联模块，用于若判定所述预设采集库中存在所述目标采集规则，则根据所述相关表格标识，确定对应的目标表单，将所述目标表单中的表格进行关联；A form association module, configured to determine a corresponding target form according to the relevant form identifier if it is determined that the target collection rule exists in the preset collection library, and associate the forms in the target form;

数据采集模块，用于根据所述数据采集路径，在所述目标表单的关联表格中获取所述目标字段集合对应的目标数据，并将所述目标数据存储至所述采集目的表。A data collection module is configured to obtain target data corresponding to the target field set in an association table of the target form according to the data collection path, and store the target data in the collection destination table.

此外，为实现上述目的，本申请还提供一种数据采集设备，所述数据采集设备包括处理器、存储器、以及存储在所述存储器上并可被所述处理器执行的计算机可读指令，其中所述计算机可读指令被所述处理器执行时，实现如上述的数据采集方法的步骤。In addition, in order to achieve the foregoing object, the present application further provides a data acquisition device, where the data acquisition device includes a processor, a memory, and computer-readable instructions stored on the memory and executable by the processor, where When the computer-readable instructions are executed by the processor, the steps of the data acquisition method as described above are implemented.

此外，为实现上述目的，本申请还提供一种计算机可读存储介质，其特征在于，所述计算机可读存储介质上存储有计算机可读指令，其中所述计算机可读指令被处理器执行时，实现如上述的数据采集方法的步骤。In addition, in order to achieve the above object, the present application further provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer-readable instructions, wherein when the computer-readable instructions are executed by a processor, To implement the steps of the data acquisition method described above.

本申请提供一种数据采集方法，即在接收到业务需求时，获取所述业务需求中的目标字段集合，并根据所述目标字段集合生成采集目的表；将所述目标字段集合与预设采集库进行匹配，判断所述预设采集库中是否存在与所述目标字段集合匹配的目标采集规则，其中，所述目标采集规则包括相关表格标识以及数据采集路径；若判定所述预设采集库中存在所述目标采集规则，则根据所述相关表格标识，确定对应的目标表单，将所述目标表单中的表格进行关联；根据所述数据采集路径，在所述目标表单的关联表格中获取所述目标字段集合对应的目标数据，并将所述目标数据存储至所述采集目的表。通过上述方式，本申请实现了目标数据的自动采集，节省了人力，提升采集效率，解决了现有采集数据技术浪费人力以及效率低下的问题。This application provides a data collection method, that is, when a business demand is received, a target field set in the business demand is obtained, and a collection purpose table is generated according to the target field set; the target field set and a preset collection are collected The database is matched to determine whether there is a target collection rule in the preset collection library that matches the target field set, wherein the target collection rule includes a related form identifier and a data collection path; if it is determined that the preset collection library If the target collection rule exists in the target form, the corresponding target form is determined according to the related form identifier, and the forms in the target form are associated; according to the data collection path, obtained in the association form of the target form The target data corresponding to the target field set is stored in the collection target table. In the above manner, the present application realizes automatic collection of target data, saves manpower, improves collection efficiency, and solves the problems of wasting manpower and low efficiency of existing data collection technologies.

附图说明BRIEF DESCRIPTION OF THE DRAWINGS

图1为本申请实施例方案中涉及的数据采集设备的硬件结构示意图；FIG. 1 is a schematic diagram of a hardware structure of a data acquisition device involved in a solution according to an embodiment of the present application; FIG.

图2为本申请数据采集方法第一实施例的流程示意图；FIG. 2 is a schematic flowchart of a first embodiment of a data collection method of this application;

图3为本申请数据采集方法第二实施例的流程示意图；FIG. 3 is a schematic flowchart of a second embodiment of a data collection method of this application; FIG.

图4为本申请数据采集方法第三实施例的流程示意图；4 is a schematic flowchart of a third embodiment of a data collection method according to the present application;

图5为本申请数据采集装置第一实施例的功能模块示意图。FIG. 5 is a schematic diagram of the functional modules of the first embodiment of the data acquisition device of the present application.

本申请目的的实现、功能特点及优点将结合实施例，参照附图做进一步说明。The implementation, functional features and advantages of the purpose of this application will be further described with reference to the embodiments and the drawings.

具体实施方式detailed description

应当理解，此处所描述的具体实施例仅仅用以解释本申请，并不用于限定本申请。It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.

本申请实施例涉及的数据采集方法主要应用于数据采集设备，该数据采集设备可以是PC、便携计算机、移动终端等具有显示和处理功能的设备。The data acquisition method according to the embodiment of the present application is mainly applied to a data acquisition device, and the data acquisition device may be a device with display and processing functions such as a PC, a portable computer, and a mobile terminal.

参照图1，图1为本申请实施例方案中涉及的数据采集设备的硬件结构示意图。本申请实施例中，数据采集设备可以包括处理器1001（例如CPU），通信总线1002，用户接口1003，网络接口1004，存储器1005。其中，通信总线1002用于实现这些组件之间的连接通信；用户接口1003可以包括显示屏（Display）、输入单元比如键盘（Keyboard）；网络接口1004可选的可以包括标准的有线接口、无线接口（如WI-FI接口）；存储器1005可以是高速RAM存储器，也可以是稳定的存储器（non-volatile memory），例如磁盘存储器，存储器1005可选的还可以是独立于前述处理器1001的存储装置。Referring to FIG. 1, FIG. 1 is a schematic diagram of a hardware structure of a data acquisition device involved in a solution according to an embodiment of the present application. In the embodiment of the present application, the data acquisition device may include a processor 1001 (such as a CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to implement connection and communication between these components. The user interface 1003 may include a display and an input unit such as a keyboard. The network interface 1004 may optionally include a standard wired interface and a wireless interface. (Such as the WI-FI interface); the memory 1005 can be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory, the memory 1005 may optionally be a storage device independent of the foregoing processor 1001.

本领域技术人员可以理解，图1中示出的硬件结构并不构成对数据采集设备的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。Those skilled in the art can understand that the hardware structure shown in FIG. 1 does not constitute a limitation on the data acquisition device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange different components.

继续参照图1，图1中作为一种计算机可读存储介质的存储器1005可以包括操作***、网络通信模块以及计算机可读指令。With continued reference to FIG. 1, the memory 1005 as a computer-readable storage medium in FIG. 1 may include an operating system, a network communication module, and computer-readable instructions.

在图1中，网络通信模块主要用于连接服务器，与服务器进行数据通信；而处理器1001可以调用存储器1005中存储的计算机可读指令，并执行本申请实施例提供的数据采集方法。In FIG. 1, the network communication module is mainly used to connect to a server and perform data communication with the server; and the processor 1001 can call computer-readable instructions stored in the memory 1005 and execute the data collection method provided in the embodiment of the present application.

本申请实施例提供了一种数据采集方法。An embodiment of the present application provides a data collection method.

参照图2，图2为本申请数据采集方法第一实施例的流程示意图。Referring to FIG. 2, FIG. 2 is a schematic flowchart of a first embodiment of a data collection method of the present application.

本实施例中，所述数据采集方法包括以下步骤：In this embodiment, the data collection method includes the following steps:

步骤S10, 在接收到业务需求时，获取所述业务需求中的目标字段集合，并根据所述目标字段集合生成采集目的表；Step S10, When receiving a business demand, obtaining a target field set in the business demand, and generating a collection destination table according to the target field set;

目前，日常工作中，业务用户经常需要大量的数据做统计分析，由于不同的用户需求有差异且需求变化太快，无法都通过报表支持，很多情况都是通过数据采集提取数据。处理人需对用户的每个数据采集做分析、编写脚本，一个采集耗费非常多人力。同时考虑到工作平衡和处理人休假的情况，一般用户的采集需求都是***根据处理人的工作量随机的分配给处理人，会出现A处理过的采集，B收到类似的采集时需要重复投入人力进行分析和脚本的编写，而且由于处理人的技能问题、有些采集逻辑非常复杂等，容易出现编写的脚本有问题，引发数据库资源浪费和异常的风险。At present, in daily work, business users often need a large amount of data for statistical analysis. Because different user needs are different and the requirements change too quickly, they cannot be supported by reports. In many cases, data is extracted through data collection. The processor needs to analyze and write scripts for each user's data collection. One collection takes a lot of manpower. At the same time, considering the work balance and the vacation of the processor, the general user's collection needs are randomly assigned to the processor according to the workload of the processor. A processed collection will appear, and B needs to repeat when receiving similar collections. Invest in manpower for analysis and script writing, and because of human skills, some collection logic is very complicated, etc., it is easy for the script to be written to have problems, causing the risk of waste of database resources and exceptions.

本实施例中，为了避免上述问题，建立一个数据采集平台，将用户的采集需求信息（即需要采集数据对应的字段信息）记录下来。处理人在根据待采集数据确定对应的字段信息时，分析源数据库中各个表单信息，确定可采集所述字段信息的目标表单。其中，优先从一个表单中获取目标数据，或者尽量降低目标数据的表单来源数量。然后获取的表单名称或者表单所在源数据库中地址，作为数据采集路径。若需要从多个表单中获取目标数据，则将所述多个表单进行关联，并根据所述数据采集路径和对应关联关系生成目标采集规则。然后将所述目标数据对应的目标字段集合与该目标采集规则对应存储至预设采集库中。具体地，在接收到用户上传的业务需求时，对所述业务需求进行解析，获取所述业务需求中的目标字段集合。然后根据所述目标字段集合生成对应的采集目的表，其中，采集目的表用户存储所述业务需求对应的目标数据。确定待采集的目标字段集合后，将所述目标字段集合中的目标字段与所述预设采集库进行匹配，从而判断所述预设采集库中是否存在所述目标字段对应的目标采集规则。In this embodiment, in order to avoid the above problems, a data collection platform is established to record the user's collection requirements information (that is, the field information corresponding to the data to be collected). When the processor determines the corresponding field information according to the data to be collected, it analyzes each form information in the source database to determine the target form that can collect the field information. Among them, the target data is preferably obtained from a form, or the number of form sources of the target data is minimized. Then obtain the form name or the address in the source database where the form is located as the data collection path. If target data needs to be obtained from multiple forms, the multiple forms are correlated, and a target collection rule is generated according to the data collection path and the corresponding association relationship. Then, the target field set corresponding to the target data and the target collection rule are correspondingly stored in a preset collection library. Specifically, when a business requirement uploaded by a user is received, the business requirement is parsed to obtain a target field set in the business requirement. Then, a corresponding collection destination table is generated according to the target field set, where the collection destination table user stores target data corresponding to the business requirements. After determining the target field set to be collected, the target fields in the target field set are matched with the preset collection library to determine whether a target collection rule corresponding to the target field exists in the preset collection library.

步骤S20，将所述目标字段集合与预设采集库进行匹配，判断所述预设采集库中是否存在与所述目标字段集合匹配的目标采集规则，其中，所述目标采集规则包括相关表格标识以及数据采集路径；Step S20: Match the target field set with a preset collection library to determine whether a target collection rule matching the target field set exists in the preset collection library, where the target collection rule includes a related form identifier And data collection paths;

本实施例中，若所述预设采集库中存在与所述目标字段集合匹配的采集规则，即表示所述预设采集库中预先存储有与所述目标字段集合相匹配的字段集合。其中，与所述目标字段集合相匹配的字段集合可以是与目标字段集合相同的字段集合，也可以是与目标字段集合类似的字段集合。与目标字段集合相同的字段集合为：有且只有与所述目标字段集合中的字段相同的字段，且字段名称均相同。与目标字段集合类似的字段集合为：有且只有与所述目标字段集合中的字段等同的字段。由于数据字段可能具有多种不同名称，预先在预设采集库中将各个字段对应的各个名称进行等同存储。等同的字段为具有不同名称但对应于同一字段的字段。然后在所述预设采集库中获取与所述目标字段集合匹配的字段集合对应的采集规则，并将该采集规则作为目标采集规则。根据所述目标采集规则，在对应的关联表单中，采集对应目标数据，并将所述目标数据存储至目的表中。如，接收到获取保单数据获取需求时，解析出所述保单数据获取需求中的保单号、保单费、保单日期、承保人、投保人以及赔偿金额等字段，并根据上述字段生成目标字段集合。将{保单号，保单费，保单日期，承保人，投保人，赔偿金额}与预设采集库中的字段集合进行比较，判断所述预设采集库中是否存在与所述{保单费，保单日期，承保人，投保人，赔偿金额}目标字段集合匹配的字段集合。若存在该匹配的字段集合，则获取该匹配的字段集合对应的采集规则，作为目标采集规则。并按照该目标采集规则获取对应数据，并存储至目的表中。In this embodiment, if a collection rule matching the target field set exists in the preset collection library, it means that the field collection that matches the target field set is stored in the preset collection library in advance. The field set matching the target field set may be the same field set as the target field set, or may be a field set similar to the target field set. The same field set as the target field set is: there is only one field that is the same as the field in the target field set, and the field names are the same. A field set similar to the target field set is: there are and only fields equivalent to the fields in the target field set. Because the data fields may have multiple different names, each name corresponding to each field is stored in the preset collection library in advance. Equivalent fields are fields that have different names but correspond to the same field. Then, a collection rule corresponding to the field set matching the target field set is obtained in the preset collection library, and the collection rule is used as the target collection rule. According to the target collection rule, corresponding target data is collected in a corresponding association form, and the target data is stored in a target table. For example, when receiving the policy data acquisition requirement, the policy number, policy fee, policy date, insurer, policyholder, and compensation amount fields in the policy data acquisition requirement are parsed, and a target field set is generated according to the above fields. Compare {Policy Number, Policy Fee, Policy Date, Underwriter, Insured, Compensation Amount} with the set of fields in the preset collection library to determine whether the preset collection database exists with the {Policy Fee, Policy Date, insurer, policyholder, compensation amount} target field set matches the field set. If the matching field set exists, a collection rule corresponding to the matched field set is obtained as a target collection rule. Corresponding data is obtained according to the target collection rule and stored in the destination table.

步骤S30，若判定所述预设采集库中存在所述目标采集规则，则根据所述相关表格标识，确定对应的目标表单，将所述目标表单中的表格进行关联；Step S30: if it is determined that the target collection rule exists in the preset collection library, determine a corresponding target form according to the related form identifier, and associate the forms in the target form;

本实施例中，若在所述预设采集库中存在所述目标采集规则，则对所述目标采集规则进行解析，获取所述目标采集规则中的相关表格标识和数据采集路径。其中，相关表格标识为需要采集该业务需求对应的目标字段集合需要调用的表格标识，该标识可以是表格名称或者表格编号。在确定所述目标字段集合对应的相关表格标识时，确定对应的目标表单，即包括可获取全部目标字段的所有表格清单，并将所述目标表单中的各个表格进行关联，以便后续进行对应字段数据的获取。其中，所述目标表单可以存储表格标识，也可以是直接获取相应的表格进行存储。In this embodiment, if the target collection rule exists in the preset collection library, the target collection rule is parsed to obtain a related form identifier and a data collection path in the target collection rule. The related form identifier is a form identifier that needs to be called to collect the target field set corresponding to the business requirement, and the identifier may be a form name or form number. When determining the related form identifier corresponding to the target field set, determine the corresponding target form, that is, include a list of all forms for which all target fields can be obtained, and associate each form in the target form for subsequent corresponding fields Acquisition of data. The target form may store a form identifier or directly obtain a corresponding form for storage.

步骤S40，根据所述数据采集路径，在所述目标表单的关联表格中获取所述目标字段集合对应的目标数据，并将所述目标数据存储至所述采集目的表。Step S40: According to the data collection path, obtain target data corresponding to the target field set in an association table of the target form, and store the target data in the collection destination table.

本实施例中，在所述目标采集规则中解析出数据采集路径。其中，数据采集路径为相关表格标识对应表格的存储地址，以及获取目标字段集合中字段的获取顺序，或者获取所述目标字段集合中字段的获取流程等。可根据所述数据采集路径中的表格存储地址，定位至对应表格，并按照所述数据采集路径中的采集流程，分别获取所述目标字段集合中的各个字段数据。并将采集到的目标字段集合对应的目标数据，根据字段格式对应存储至所述采集目的表中，并将所述采集目的表反馈至对应用户。In this embodiment, a data collection path is parsed in the target collection rule. The data collection path is the storage address of the corresponding form identification table, the order of obtaining the fields in the target field set, or the process of obtaining the fields in the target field set. According to the form storage address in the data collection path, the corresponding form can be located, and each field data in the target field set can be obtained according to the collection process in the data collection path. The target data corresponding to the collected target field set is correspondingly stored in the collection purpose table according to the field format, and the collection purpose table is fed back to the corresponding user.

本实施例提供一种数据采集方法，即在接收到业务需求时，获取所述业务需求中的目标字段集合，并根据所述目标字段集合生成采集目的表；将所述目标字段集合与预设采集库进行匹配，判断所述预设采集库中是否存在与所述目标字段集合匹配的目标采集规则，其中，所述目标采集规则包括相关表格标识以及数据采集路径；若判定所述预设采集库中存在所述目标采集规则，则根据所述相关表格标识，确定对应的目标表单，将所述目标表单中的表格进行关联；根据所述数据采集路径，在所述目标表单的关联表格中获取所述目标字段集合对应的目标数据，并将所述目标数据存储至所述采集目的表。通过上述方式，本申请实现了目标数据的自动采集，节省了人力，提升采集效率，解决了现有采集数据技术浪费人力以及效率低下的问题。This embodiment provides a data collection method, that is, when a business demand is received, a target field set in the business demand is obtained, and a collection purpose table is generated according to the target field set; the target field set and a preset The collection database is matched to determine whether there is a target collection rule in the preset collection library that matches the target field set, wherein the target collection rule includes a related form identifier and a data collection path; if it is determined that the preset collection If the target collection rule exists in the database, the corresponding target form is determined according to the related form identifier, and the tables in the target form are associated; according to the data collection path, in the associated form of the target form Acquiring target data corresponding to the target field set, and storing the target data in the collection destination table. In the above manner, the present application realizes automatic collection of target data, saves manpower, improves collection efficiency, and solves the problems of wasting manpower and low efficiency of existing data collection technologies.

参照图3，图3为本申请数据采集方法第二实施例的流程示意图。Referring to FIG. 3, FIG. 3 is a schematic flowchart of a second embodiment of a data collection method of the present application.

基于上述图2所示实施例，本实施例中，所述步骤S20包括：Based on the embodiment shown in FIG. 2, in this embodiment, the step S20 includes:

步骤S21，将所述目标字段集合中的各个目标字段与所述预设采集库中的预设字段集合进行匹配，并计算所述目标字段集合与所述预设字段集合的匹配度；Step S21: Match each target field in the target field set with a preset field set in the preset collection library, and calculate a degree of matching between the target field set and the preset field set;

步骤S22，根据所述匹配度，判断所述预设采集库中是否存在与所述目标字段集合匹配的目标采集规则，其中，与所述目标字段集合的匹配度不小于预设阈值的预设字段集合对应的采集规则为所述目标采集规则。In step S22, it is determined whether a target collection rule matching the target field set exists in the preset collection library according to the matching degree, and the matching degree with the target field set is not less than a preset preset value. The collection rule corresponding to the field set is the target collection rule.

本实施例中，计算所述预设采集库中各个预设字段集合与所述业务需求对应的目标字段集合对应的匹配度。具体步骤为：依次获取所述预设采集库中的各个预设字段集合，并将所述目标字段集合中的各个目标字段与预设字段集合中的各个字段进行匹配，且一个预设字段集合为一匹配单位。获取所述目标字段集合与各个预设字段集合的相匹配字段个数，并将所述相匹配字段个数除以所述目标字段集合中的字段个数，即得到所述目标字段集合与各个预设字段集合的匹配度。设定一预设阈值即匹配度阈值，如100%，若存在等于所述预设阈值的匹配度，即判定至少存在某个预设字段集合与所述目标字段集合中的各个目标字段均匹配。若不存在等于所述预设阈值的匹配度，即判定所述预设采集库中的预设采集集合均不符合所述目标字段集合。具体实施例中，进一步获取所述目标字段集合以及预设字段集合对应的字段数目，在存在多个符合条件的预设字段集合时，优先获取与所述目标字段集合字段数目相同的预设字段集合作为对应字段集合，从而增加目标采集规则的精确度。In this embodiment, a matching degree corresponding to each preset field set in the preset collection library and a target field set corresponding to the service requirement is calculated. The specific steps are: sequentially obtaining each preset field set in the preset collection library, and matching each target field in the target field set with each field in the preset field set, and a preset field set Is a matching unit. Obtain the number of matching fields of the target field set and each preset field set, and divide the number of matching fields by the number of fields in the target field set to obtain the target field set and each The matching degree of the preset field set. Set a preset threshold, that is, the matching degree threshold, such as 100%. If there is a matching degree equal to the preset threshold, it is determined that at least one preset field set matches each target field in the target field set. . If there is no matching degree equal to the preset threshold, it is determined that none of the preset collection sets in the preset collection library meets the target field set. In a specific embodiment, the target field set and the number of fields corresponding to the preset field set are further obtained. When there are multiple preset field sets that meet the conditions, the preset fields with the same number of fields as the target field set are preferentially obtained. The set is used as the corresponding field set, thereby increasing the accuracy of the target collection rule.

进一步地，步骤S20之后，还包括：Further, after step S20, the method further includes:

本实施例中，若判定所述预设采集库中不存在与所述目标字段集合相符的预设字段集合，即判定不存在与所述目标字段集合匹配的所述目标采集规则。为了便于技术人员参考，可获取在所述预设采集库中获取与所述目标字段集合的匹配度最大的预设字段集合，并将该预设字段集合设置为相关字段集合。该相关字段集合具有与目标字段集合相近的采集规则，在所述预设采集库中获取所述相关字段集合对应的采集规则，作为相关采集规则，并将所述相关采集规则推送至所述用户进行参考。该相关字段集合可以为具有所述目标字段集合中的大部分字段的字段集合。In this embodiment, if it is determined that there is no preset field set in the preset collection library that matches the target field set, it is determined that there is no target collection rule that matches the target field set. In order to facilitate reference by a technician, a preset field set that has the highest degree of matching with the target field set can be obtained in the preset collection library, and the preset field set is set as a related field set. The related field set has a collection rule similar to the target field set. A collection rule corresponding to the related field set is obtained in the preset collection library as a related collection rule, and the related collection rule is pushed to the user. For reference. The related field set may be a field set with most fields in the target field set.

参照图4，图4为本申请数据采集方法第三实施例的流程示意图。Referring to FIG. 4, FIG. 4 is a schematic flowchart of a third embodiment of a data collection method of the present application.

基于上述图3所示实施例，本实施例中，所述步骤S30包括：Based on the embodiment shown in FIG. 3 above, in this embodiment, the step S30 includes:

步骤S31，若所述预设采集库中存在所述目标采集规则，则获取所述目标采集规则的规则数目，并判断所述规则数目是否唯一；Step S31: if the target collection rule exists in the preset collection database, obtain the number of rules of the target collection rule, and determine whether the number of rules is unique;

步骤S32，若所述规则数目不唯一，则获取所述目标采集规则中的待定采集规则以及对应的匹配度；Step S32: if the number of rules is not unique, obtain a pending collection rule and a corresponding matching degree among the target collection rules;

步骤S33，根据所述匹配度大小，将所述待定采集规则进行排序，生成并显示对应的待定采集规则列表；Step S33: Sort the pending collection rules according to the matching degree to generate and display a corresponding list of pending collection rules;

步骤S34，基于所述待定采集规则列表，接收用户选择操作触发的选择指令，并根据所述选择指令，在所述待定采集规则列表中确定目标采集规则。Step S34: Receive a selection instruction triggered by a user selection operation based on the pending collection rule list, and determine a target collection rule in the pending collection rule list according to the selection instruction.

进一步地，所述步骤S30还包括：Further, the step S30 further includes:

步骤S35，若所述规则数目唯一，则根据所述目标采集规则中的相关表格标识，确定对应的目标表单，将所述目标表单中的表格进行关联。In step S35, if the number of rules is unique, a corresponding target form is determined according to a related form identifier in the target collection rule, and the forms in the target form are associated.

本实施例中，为了避免一个预设字段集合对应多个采集规则，在确定所述预设采集库中存在所述目标采集规则后，进一步获取所述目标采集规则的规则数目。判断所述目标采集规则的规则数目是否唯一，若所述规则数目不唯一，即表示所述预设采集库中存在多个采集规则可获取所述目标字段集合中的目标数据。根据上述实施例中所述目标字段集合与各个预设字段集合的匹配度大小，如按照由大到小的顺序，将符合要求的目标采集规则中的采集规则进行排序，然后进一步根据所述排序和所述待定采集规则生成对应的待定采集规则列表，并通过对应终端进行显示，以便用户根据需用进行选择。在接收到用户选择操作触发的选择指令时，获取所述选择指令中的采集规则标识，并根据所述采集规则标识将该采集规则设定为目标采集规则。若所述规则数目唯一，即所述目标采集规则即为唯一选择，根据所述目标采集规则确定对应的相关表格标识，建立目标表单，并将所述目标表单中的各个表格进行关联，便于后续采集数据。In this embodiment, in order to avoid that one preset field set corresponds to multiple collection rules, after determining that the target collection rule exists in the preset collection database, the number of rules of the target collection rule is further obtained. Determine whether the number of rules of the target collection rule is unique. If the number of rules is not unique, it means that there are multiple collection rules in the preset collection library to obtain target data in the target field set. According to the matching degree between the target field set and each preset field set in the foregoing embodiment, for example, the collection rules in the target collection rules that meet the requirements are sorted in descending order, and then further based on the sort A list of pending collection rules is generated corresponding to the pending collection rules, and displayed on the corresponding terminal, so that the user can select according to needs. When a selection instruction triggered by a user selection operation is received, a collection rule identifier in the selection instruction is obtained, and the collection rule is set as a target collection rule according to the collection rule identifier. If the number of rules is unique, that is, the target collection rule is the only choice. According to the target collection rule, the corresponding related form identifier is determined, a target form is established, and each form in the target form is associated for subsequent follow-up. Data collection.

进一步地，所述步骤S33之后，还可以包括：Further, after step S33, the method may further include:

本实施例中，优先从较少的表格中获取目标数据，即在所述规则数目不唯一时，获取所述目标采集规则中的各个采集规则，作为待选采集规则。然后获取各个待选采集规则中的表格标识，即一个表格标识对应一个表格，通过所述表格标识可确定所述待选采集规则中的表格数量。具体实施例中，可根据所述表格数量的大小，如由小到大，将各个待选采集规则进行排序，并选取表格数量最小的待选采集规则，作为目标采集规则，以提高数据获取效率。In this embodiment, the target data is preferentially obtained from fewer tables, that is, when the number of rules is not unique, the target is obtained. Each collection rule in the collection rule is used as a candidate collection rule. Then, the form identifier in each candidate collection rule is obtained, that is, one form identifier corresponds to one table, and the number of tables in the candidate collection rule can be determined by the form identifier. In a specific embodiment, the candidate collection rules can be sorted according to the size of the number of tables, such as from small to large, and the candidate collection rule with the smallest number of tables is selected as a target collection rule to improve data acquisition efficiency. .

进一步地，所述数据采集方法还包括：Further, the data collection method further includes:

本实施例中，该预设采集库为根据数据采集的历史记录，即将用户的采集需求、对应编写的脚本以及对应的采集字段信息均记录下来，然后对上述记录数据进行分析、标签以及归类而形成的。因此，若在所述预设采集库中不存在与所述目标字段集合对应的目标采集规则，即表示数据采集平台中未存储有对应的采集规则。可将所述目标字段集合或者将整个业务需求上报至预设处理端，该预设处理端为对应的技术人员所使用的处理端，以将所述目标字段集合上报至对应处理***进行人工编写对应的采集规则。具体实施例中，所述数据采集平台可将与所述目标字段集合匹配度最大的相关字段集合以及对应的相关采集规则一同上报至所述预设处理端，以便所述技术人员进行参考。然后在接收到所述预设处理端反馈的采集规则时，将所述反馈的采集规则作为目标采集规则，根据所述目标采集规则采集对应的目标数据，并将存储有目的数据的采集目的表反馈至用户。然后将所述目标字段集合以及所述预设处理端反馈的采集规则对应存储至所述预设采集库中，从而更新所述预设采集库中的存储数据。In this embodiment, the preset collection library is based on the historical records of data collection, that is, to record the user's collection requirements, correspondingly written scripts, and corresponding collection field information, and then analyze, label, and classify the recorded data. Formed. Therefore, if there is no target collection rule corresponding to the target field set in the preset collection library, it means that no corresponding collection rule is stored in the data collection platform. The target field set or the entire business requirement may be reported to a preset processing end, which is a processing end used by a corresponding technician to report the target field set to a corresponding processing system for manual writing Corresponding collection rules. In a specific embodiment, the data collection platform may report the related field set that has the highest degree of matching with the target field set and the corresponding related collection rule to the preset processing end for reference by the technician. Then, when receiving the collection rule feedback from the preset processing end, the feedback collection rule is used as the target collection rule, corresponding target data is collected according to the target collection rule, and the collection purpose table storing the target data is stored. Feedback to users. Then, the target field set and the collection rule feedback from the preset processing end are correspondingly stored in the preset collection library, thereby updating the stored data in the preset collection library.

此外，本申请实施例还提供一种数据采集装置。In addition, an embodiment of the present application further provides a data acquisition device.

参照图5，图5为本申请数据采集装置第一实施例的功能模块示意图。Referring to FIG. 5, FIG. 5 is a schematic diagram of functional modules of a first embodiment of a data acquisition device of the present application.

本实施例中，所述数据采集装置包括：In this embodiment, the data acquisition device includes:

表格生成模块10，用于在接收到业务需求时，获取所述业务需求中的目标字段集合，并根据所述目标字段集合生成采集目的表；The form generating module 10 is configured to obtain a target field set in the business demand when a business demand is received, and generate a collection destination table according to the target field set;

规则判断模块20，用于将所述目标字段集合与预设采集库进行匹配，判断所述预设采集库中是否存在与所述目标字段集合匹配的目标采集规则，其中，所述目标采集规则包括相关表格标识以及数据采集路径；A rule judgment module 20 is configured to match the target field set with a preset collection library, and determine whether a target collection rule matching the target field set exists in the preset collection library, wherein the target collection rule Including relevant form identification and data collection path;

表格关联模块30，用于若判定所述预设采集库中存在所述目标采集规则，则根据所述相关表格标识，确定对应的目标表单，将所述目标表单中的表格进行关联；A form association module 30, configured to determine a corresponding target form according to the related form identifier if it is determined that the target collection rule exists in the preset collection library, and associate the forms in the target form;

数据采集模块40，用于根据所述数据采集路径，在所述目标表单的关联表格中获取所述目标字段集合对应的目标数据，并将所述目标数据存储至所述采集目的表。A data collection module 40 is configured to obtain target data corresponding to the target field set in an association table of the target form according to the data collection path, and store the target data in the collection destination table.

进一步地，所述规则判断模块20具体包括：Further, the rule judgment module 20 specifically includes:

匹配度计算单元，用于将所述目标字段集合中的各个目标字段与所述预设采集库中的预设字段集合进行匹配，并计算所述目标字段集合与所述预设字段集合的匹配度；A matching degree calculation unit, configured to match each target field in the target field set with a preset field set in the preset collection library, and calculate a match between the target field set and the preset field set degree;

规则判断单元，用于根据所述匹配度，判断所述预设采集库中是否存在与所述目标字段集合匹配的目标采集规则，其中，与所述目标字段集合的匹配度不小于预设阈值的预设字段集合对应的采集规则为所述目标采集规则。A rule judging unit, configured to determine, according to the matching degree, whether a target collection rule matching the target field set exists in the preset collection library, and the matching degree with the target field set is not less than a preset threshold The collection rule corresponding to the preset field set is the target collection rule.

进一步的，所述数据采集装置还包括：Further, the data acquisition device further includes:

字段获取模块，用于若判定所述预设采集库中不存在所述目标采集规则，则获取与所述目标字段集合的匹配度最大的预设字段集合，作为相关字段集合；A field obtaining module, configured to determine, if it is determined that the target collection rule does not exist in the preset collection library, a preset field set that has the highest degree of matching with the target field set as a related field set;

规则推送模块，用于获取所述相关字段集合对应的采集规则，作为相关采集规则，并将所述相关采集规则推送至当前终端，以便用户参考。A rule pushing module is configured to obtain a collection rule corresponding to the related field set as a related collection rule, and push the related collection rule to a current terminal for reference by a user.

进一步地，所述表格关联模块30包括：Further, the table association module 30 includes:

数目判断单元，用于若所述预设采集库中存在所述目标采集规则，则获取所述目标采集规则的规则数目，并判断所述规则数目是否唯一；A number judging unit, configured to, if the target collection rule exists in the preset collection database, obtain the number of rules of the target collection rule, and determine whether the number of rules is unique;

第一规则获取单元，用于若所述规则数目不唯一，则获取所述目标采集规则中的待定采集规则以及对应的匹配度；A first rule obtaining unit, configured to obtain, if the number of rules is not unique, a pending collection rule in the target collection rule and a corresponding matching degree;

规则排序单元，用于根据所述匹配度大小，将所述待定采集规则进行排序，生成并显示对应的待定采集规则列表；A rule sorting unit, configured to sort the pending collection rules according to the matching degree to generate and display a corresponding list of pending collection rules;

规则选择单元，用于基于所述待定采集规则列表，接收用户选择操作触发的选择指令，并根据所述选择指令，在所述待定采集规则列表中确定目标采集规则。A rule selection unit is configured to receive a selection instruction triggered by a user selection operation based on the pending collection rule list, and determine a target collection rule in the pending collection rule list according to the selection instruction.

表单确定单元，用于若所述规则数目唯一，则根据所述目标采集规则中的相关表格标识，确定对应的目标表单，将所述目标表单中的表格进行关联。A form determination unit is configured to determine a corresponding target form according to a related form identifier in the target collection rule if the number of rules is unique, and associate the forms in the target form.

进一步地，所述数据采集装置还包括：Further, the data acquisition device further includes:

第二规则获取单元，用于若所述规则数目不唯一，则获取所述目标采集规则中的各个采集规则，作为待选采集规则；A second rule obtaining unit, configured to obtain each collection rule in the target collection rule as a candidate collection rule if the number of rules is not unique;

数量确定单元，用于根据所述待选采集规则中表格标识，确定所述各个待选采集规则对应的表格数量，并获取表格数量最小的待选采集规则，作为目标采集规则。A quantity determining unit is configured to determine the number of tables corresponding to each of the candidate collection rules according to the table identifier in the candidate collection rule, and obtain the candidate collection rule with the smallest number of tables as a target collection rule.

上报单元，用于若判定所述预设采集库中不存在所述目标采集规则，则将所述目标字段集合上报至预设处理端；A reporting unit, configured to report the target field set to a preset processing end if it is determined that the target collection rule does not exist in the preset collection library;

更新单元，用于在接收到所述预设处理端反馈的采集规则时，将所述预设处理端反馈的采集规则作为目标采集规则，并将所述目标字段集合和所述目标采集规则对应存储至所述预设采集库，以更新所述预设采集库。An update unit is configured to: when receiving the collection rule feedback from the preset processing end, use the collection rule feedback from the preset processing end as a target collection rule, and associate the target field set with the target collection rule Store to the preset acquisition library to update the preset acquisition library.

其中，上述数据采集装置中各个模块与上述数据采集方法实施例中各步骤相对应，其功能和实现过程在此处不再一一赘述。Each module in the foregoing data acquisition device corresponds to each step in the foregoing embodiment of the data acquisition method, and its functions and implementation processes are not repeated here one by one.

此外，本申请实施例还提供一种计算机可读存储介质。In addition, an embodiment of the present application further provides a computer-readable storage medium.

本申请计算机可读存储介质上存储有计算机可读指令，其中，所述计算机可读存储介质可以为非易失性可读存储介质，所述计算机可读指令被处理器执行时，实现如上述的数据采集方法的步骤。The computer-readable storage medium of the present application stores computer-readable instructions, wherein the computer-readable storage medium may be a non-volatile readable storage medium, and when the computer-readable instructions are executed by a processor, the implementation is as described above. Steps of the data acquisition method.

其中，计算机可读指令被执行时所实现的方法可参照本申请数据采集方法的各个实施例，此处不再赘述。For the method implemented when the computer-readable instructions are executed, reference may be made to various embodiments of the data collection method of this application, and details are not described herein again.

需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者***不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者***所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、方法、物品或者***中还存在另外的相同要素。It should be noted that, in this article, the terms "including", "including" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, It also includes other elements not explicitly listed, or elements inherent to such a process, method, article, or system. Without more restrictions, an element limited by the sentence "including a ..." does not exclude the existence of other identical elements in the process, method, article, or system that includes the element.

上述本申请实施例序号仅仅为了描述，不代表实施例的优劣。The above-mentioned serial numbers of the embodiments of the present application are merely for description, and do not represent the superiority or inferiority of the embodiments.

通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在如上所述的一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端设备(可以是手机，计算机，服务器，空调器，或者网络设备等)执行本申请各个实施例所述的方法。Through the description of the above embodiments, those skilled in the art can clearly understand that the methods in the above embodiments can be implemented by means of software plus a necessary universal hardware platform, and of course, also by hardware, but in many cases the former is better. Implementation. Based on such an understanding, the technical solution of this application that is essentially or contributes to the existing technology can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM) as described above. , Magnetic disk, optical disc), including a number of instructions to enable a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in the embodiments of this application.

以上仅为本申请的优选实施例，并非因此限制本申请的专利范围，凡是利用本申请说明书及附图内容所作的等效结构或等效流程变换，或直接或间接运用在其他相关的技术领域，均同理包括在本申请的专利保护范围内。The above are only preferred embodiments of the present application, and thus do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the contents of the specification and drawings of the present application, or directly or indirectly used in other related technical fields Are included in the scope of patent protection of this application.

Claims

一种数据采集方法，其特征在于，所述数据采集方法包括以下步骤： A data collection method, characterized in that the data collection method includes the following steps:

在接收到业务需求时，获取所述业务需求中的目标字段集合，并根据所述目标字段集合生成采集目的表；When receiving a business demand, obtaining a target field set in the business demand, and generating a collection destination table according to the target field set;

将所述目标字段集合与预设采集库进行匹配，判断所述预设采集库中是否存在与所述目标字段集合匹配的目标采集规则，其中，所述目标采集规则包括相关表格标识以及数据采集路径；Matching the target field set with a preset collection library to determine whether there is a target collection rule matching the target field set in the preset collection library, wherein the target collection rule includes a related form identifier and data collection path;

若判定所述预设采集库中存在所述目标采集规则，则根据所述相关表格标识，确定对应的目标表单，将所述目标表单中的表格进行关联；If it is determined that the target collection rule exists in the preset collection library, determining a corresponding target form according to the related form identifier, and correlating the forms in the target form;

根据所述数据采集路径，在所述目标表单的关联表格中获取所述目标字段集合对应的目标数据，并将所述目标数据存储至所述采集目的表。According to the data collection path, obtain target data corresponding to the target field set in an association table of the target form, and store the target data in the collection destination table.
如权利要求1所述的数据采集方法，其特征在于，所述将所述目标字段集合与预设采集库进行匹配，判断所述预设采集库中是否存在与所述目标字段集合匹配的目标采集规则，其中，所述目标采集规则包括对应关联关系以及数据采集路径的步骤包括：The data collection method according to claim 1, wherein the matching of the target field set with a preset collection library determines whether there is a target in the preset collection library that matches the target field collection. A collection rule, wherein the step of the target collection rule including a corresponding association relationship and a data collection path includes:

将所述目标字段集合中的各个目标字段与所述预设采集库中的预设字段集合进行匹配，并计算所述目标字段集合与所述预设字段集合的匹配度；Matching each target field in the target field set with a preset field set in the preset collection library, and calculating a degree of matching between the target field set and the preset field set;

根据所述匹配度，判断所述预设采集库中是否存在与所述目标字段集合匹配的目标采集规则，其中，与所述目标字段集合的匹配度不小于预设阈值的预设字段集合对应的采集规则为所述目标采集规则。Determining, according to the matching degree, whether a target collection rule matching the target field set exists in the preset collection library, wherein the preset field set corresponding to the target field set is not less than a preset threshold value The collection rule is the target collection rule.
如权利要求2所述的数据采集方法，其特征在于，所述根据所述匹配度，判断所述预设采集库中是否存在与所述目标字段集合匹配的目标采集规则，其中，与所述目标字段集合的匹配度不小于预设阈值的预设字段集合对应的采集规则为所述目标采集规则的步骤之后，还包括：The data collection method according to claim 2, wherein, according to the matching degree, determining whether a target collection rule matching the target field set exists in the preset collection library, wherein After the step that the target field set has a matching degree that is not less than a preset threshold, the collection rule corresponding to the preset field set is the target collection rule, further including:

若判定所述预设采集库中不存在所述目标采集规则，则获取与所述目标字段集合的匹配度最大的预设字段集合，作为相关字段集合；If it is determined that the target collection rule does not exist in the preset collection library, obtaining a preset field set that has the greatest degree of matching with the target field set as a related field set;

获取所述相关字段集合对应的采集规则，作为相关采集规则，并将所述相关采集规则推送至当前终端，以便用户参考。Acquire a collection rule corresponding to the related field set as a related collection rule, and push the related collection rule to a current terminal for reference by a user.
如权利要求2所述的数据采集方法，其特征在于，所述若判定所述预设采集库中存在所述目标采集规则，则根据所述相关表格标识，确定对应的目标表单，将所述目标表单中的表格进行关联的步骤包括：The data collection method according to claim 2, wherein if it is determined that the target collection rule exists in the preset collection library, determining a corresponding target form according to the related form identifier, and converting the target form The steps to associate the tables in the target form include:

若所述预设采集库中存在所述目标采集规则，则获取所述目标采集规则的规则数目，并判断所述规则数目是否唯一；If the target collection rule exists in the preset collection database, obtaining the number of rules of the target collection rule, and determining whether the number of rules is unique;

若所述规则数目唯一，则根据所述目标采集规则中的相关表格标识，确定对应的目标表单，将所述目标表单中的表格进行关联。If the number of rules is unique, a corresponding target form is determined according to a related form identifier in the target collection rule, and the forms in the target form are associated.
如权利要求4所述的数据采集方法，其特征在于，所述若所述预设采集库中存在所述目标采集规则，则获取所述目标采集规则的规则数目，并判断所述规则数目是否唯一的步骤之后，还包括：The data collection method according to claim 4, wherein if the target collection rule exists in the preset collection library, obtaining the number of rules of the target collection rule and determining whether the number of rules is After the only step, it also includes:

若所述规则数目不唯一，则获取所述目标采集规则中的待定采集规则以及对应的匹配度；If the number of rules is not unique, obtaining a pending collection rule and a corresponding matching degree among the target collection rules;

根据所述匹配度大小，将所述待定采集规则进行排序，生成并显示对应的待定采集规则列表；Sorting the pending collection rules according to the matching degree to generate and display a corresponding list of pending collection rules;

基于所述待定采集规则列表，接收用户选择操作触发的选择指令，并根据所述选择指令，在所述待定采集规则列表中确定目标采集规则。 Based on the pending collection rule list, a selection instruction triggered by a user selection operation is received, and a target collection rule is determined in the pending collection rule list according to the selection instruction. Ranch
如权利要求4所述的数据采集方法，其特征在于，所述若所述预设采集库中存在所述目标采集规则，则获取所述目标采集规则的规则数目，并判断所述规则数目是否唯一的步骤之后，还包括：The data collection method according to claim 4, wherein if the target collection rule exists in the preset collection library, obtaining the number of rules of the target collection rule and determining whether the number of rules is After the only step, it also includes:

若所述规则数目不唯一，则获取所述目标采集规则中的各个采集规则，作为待选采集规则；If the number of rules is not unique, obtaining each collection rule in the target collection rule as a candidate collection rule;

根据所述待选采集规则中表格标识，确定所述各个待选采集规则对应的表格数量，并获取表格数量最小的待选采集规则，作为目标采集规则。According to the table identifier in the candidate collection rule, the number of tables corresponding to each candidate collection rule is determined, and the candidate collection rule with the smallest number of tables is obtained as the target collection rule.
如权利要求1所述的数据采集方法，其特征在于，所述将所述目标字段集合与预设采集库进行匹配，判断所述预设采集库中是否存在与所述目标字段集合匹配的目标采集规则，其中，所述目标采集规则包括相关表格标识以及数据采集路径的步骤之后，还包括：The data collection method according to claim 1, wherein the matching of the target field set with a preset collection library determines whether there is a target in the preset collection library that matches the target field collection. A collection rule, where the target collection rule includes a step of identifying a related table and a data collection path, and further includes:

若判定所述预设采集库中不存在所述目标采集规则，则将所述目标字段集合上报至预设处理端；If it is determined that the target collection rule does not exist in the preset collection library, report the target field set to a preset processing end;

在接收到所述预设处理端反馈的采集规则时，将所述预设处理端反馈的采集规则作为目标采集规则，并将所述目标字段集合和所述目标采集规则对应存储至所述预设采集库，以更新所述预设采集库。When receiving the collection rule feedback from the preset processing end, the collection rule feedback from the preset processing end is used as the target collection rule, and the target field set and the target collection rule are correspondingly stored in the preset A collection library is set to update the preset collection library.
一种数据采集装置，其特征在于，所述数据采集装置包括： A data acquisition device, characterized in that the data acquisition device includes:

表格生成模块，用于在接收到业务需求时，获取所述业务需求中的目标字段集合，并根据所述目标字段集合生成采集目的表；A form generating module, configured to obtain a target field set in the business demand when a business demand is received, and generate a collection destination table according to the target field set;

规则判断模块，用于将所述目标字段集合与预设采集库进行匹配，判断所述预设采集库中是否存在与所述目标字段集合匹配的目标采集规则，其中，所述目标采集规则包括相关表格标识以及数据采集路径；A rule judgment module is configured to match the target field set with a preset collection library, and determine whether a target collection rule matching the target field set exists in the preset collection library, where the target collection rule includes Relevant form identification and data collection path;

表格关联模块，用于若判定所述预设采集库中存在所述目标采集规则，则根据所述相关表格标识，确定对应的目标表单，将所述目标表单中的表格进行关联；A form association module, configured to determine a corresponding target form according to the relevant form identifier if it is determined that the target collection rule exists in the preset collection library, and associate the forms in the target form;

数据采集模块，用于根据所述数据采集路径，在所述目标表单的关联表格中获取所述目标字段集合对应的目标数据，并将所述目标数据存储至所述采集目的表。A data collection module is configured to obtain target data corresponding to the target field set in an association table of the target form according to the data collection path, and store the target data in the collection destination table.
如权利要求8所述的数据采集装置，其特征在于，所述规则判断模块包括：The data acquisition device according to claim 8, wherein the rule judgment module comprises:

匹配度计算单元，用于将所述目标字段集合中的各个目标字段与所述预设采集库中的预设字段集合进行匹配，并计算所述目标字段集合与所述预设字段集合的匹配度；A matching degree calculation unit, configured to match each target field in the target field set with a preset field set in the preset collection library, and calculate a match between the target field set and the preset field set degree;

规则判断单元，用于根据所述匹配度，判断所述预设采集库中是否存在与所述目标字段集合匹配的目标采集规则，其中，与所述目标字段集合的匹配度不小于预设阈值的预设字段集合对应的采集规则为所述目标采集规则。A rule judging unit, configured to determine, according to the matching degree, whether a target collection rule matching the target field set exists in the preset collection library, and the matching degree with the target field set is not less than a preset threshold The collection rule corresponding to the preset field set is the target collection rule.
如权利要求8所述的数据采集装置，其特征在于，所述数据采集装置还包括：The data acquisition device according to claim 8, wherein the data acquisition device further comprises:

字段获取模块，用于若判定所述预设采集库中不存在所述目标采集规则，则获取与所述目标字段集合的匹配度最大的预设字段集合，作为相关字段集合；A field obtaining module, configured to determine, if it is determined that the target collection rule does not exist in the preset collection library, a preset field set that has the highest degree of matching with the target field set as a related field set;

规则推送模块，用于获取所述相关字段集合对应的采集规则，作为相关采集规则，并将所述相关采集规则推送至当前终端，以便用户参考。A rule pushing module is configured to obtain a collection rule corresponding to the related field set as a related collection rule, and push the related collection rule to a current terminal for reference by a user.
如权利要求8所述的数据采集装置，其特征在于，所述表格关联模块包括：The data acquisition device according to claim 8, wherein the form association module comprises:

数目判断单元，用于若所述预设采集库中存在所述目标采集规则，则获取所述目标采集规则的规则数目，并判断所述规则数目是否唯一；A number judging unit, configured to, if the target collection rule exists in the preset collection database, obtain the number of rules of the target collection rule, and determine whether the number of rules is unique;

第一规则获取单元，用于若所述规则数目不唯一，则获取所述目标采集规则中的待定采集规则以及对应的匹配度；A first rule obtaining unit, configured to obtain, if the number of rules is not unique, a pending collection rule in the target collection rule and a corresponding matching degree;

规则排序单元，用于根据所述匹配度大小，将所述待定采集规则进行排序，生成并显示对应的待定采集规则列表。A rule sorting unit is configured to sort the pending collection rules according to the matching degree, and generate and display a corresponding list of pending collection rules.
如权利要求8所述的数据采集装置，其特征在于，所述表格关联模块包括：The data acquisition device according to claim 8, wherein the form association module comprises:

规则选择单元，用于基于所述待定采集规则列表，接收用户选择操作触发的选择指令，并根据所述选择指令，在所述待定采集规则列表中确定目标采集规则。A rule selection unit is configured to receive a selection instruction triggered by a user selection operation based on the pending collection rule list, and determine a target collection rule in the pending collection rule list according to the selection instruction.

表单确定单元，用于若所述规则数目唯一，则根据所述目标采集规则中的相关表格标识，确定对应的目标表单，将所述目标表单中的表格进行关联。A form determination unit is configured to determine a corresponding target form according to a related form identifier in the target collection rule if the number of rules is unique, and associate the forms in the target form.
一种数据采集设备，其特征在于，所述数据采集设备包括处理器、存储器、以及存储在所述存储器上并可被所述处理器执行的计算机可读指令，其中所述计算机可读指令被所述处理器执行时，实现如权利要求1所述的数据采集方法的步骤。A data acquisition device, characterized in that the data acquisition device includes a processor, a memory, and computer-readable instructions stored on the memory and executable by the processor, wherein the computer-readable instructions are When the processor executes, the steps of the data acquisition method according to claim 1 are implemented.
一种数据采集设备，其特征在于，所述数据采集设备包括处理器、存储器、以及存储在所述存储器上并可被所述处理器执行的计算机可读指令，其中所述计算机可读指令被所述处理器执行时，实现如权利要求2所述的数据采集方法的步骤。A data acquisition device, characterized in that the data acquisition device includes a processor, a memory, and computer-readable instructions stored on the memory and executable by the processor, wherein the computer-readable instructions are When the processor executes, the steps of the data acquisition method according to claim 2 are implemented.
一种数据采集设备，其特征在于，所述数据采集设备包括处理器、存储器、以及存储在所述存储器上并可被所述处理器执行的计算机可读指令，其中所述计算机可读指令被所述处理器执行时，实现如权利要求3所述的数据采集方法的步骤。A data acquisition device, characterized in that the data acquisition device includes a processor, a memory, and computer-readable instructions stored on the memory and executable by the processor, wherein the computer-readable instructions are When the processor executes, the steps of the data collection method according to claim 3 are implemented.
一种数据采集设备，其特征在于，所述数据采集设备包括处理器、存储器、以及存储在所述存储器上并可被所述处理器执行的计算机可读指令，其中所述计算机可读指令被所述处理器执行时，实现如权利要求4所述的数据采集方法的步骤。A data acquisition device, characterized in that the data acquisition device includes a processor, a memory, and computer-readable instructions stored on the memory and executable by the processor, wherein the computer-readable instructions are When the processor executes, the steps of the data acquisition method according to claim 4 are implemented.
一种计算机可读存储介质，其特征在于，所述计算机可读存储介质上存储有计算机可读指令，其中所述计算机可读指令被处理器执行时，实现如权利要求1所述的数据采集方法的步骤。A computer-readable storage medium, characterized in that computer-readable instructions are stored on the computer-readable storage medium, wherein when the computer-readable instructions are executed by a processor, the data collection according to claim 1 is implemented Method steps.
一种计算机可读存储介质，其特征在于，所述计算机可读存储介质上存储有计算机可读指令，其中所述计算机可读指令被处理器执行时，实现如权利要求2所述的数据采集方法的步骤。A computer-readable storage medium, characterized in that computer-readable instructions are stored on the computer-readable storage medium, wherein when the computer-readable instructions are executed by a processor, the data collection according to claim 2 is implemented Method steps.
一种计算机可读存储介质，其特征在于，所述计算机可读存储介质上存储有计算机可读指令，其中所述计算机可读指令被处理器执行时，实现如权利要求3所述的数据采集方法的步骤。A computer-readable storage medium, characterized in that computer-readable instructions are stored on the computer-readable storage medium, wherein when the computer-readable instructions are executed by a processor, the data collection according to claim 3 is implemented Method steps.
一种计算机可读存储介质，其特征在于，所述计算机可读存储介质上存储有计算机可读指令，其中所述计算机可读指令被处理器执行时，实现如权利要求4所述的数据采集方法的步骤。A computer-readable storage medium, characterized in that computer-readable instructions are stored on the computer-readable storage medium, wherein when the computer-readable instructions are executed by a processor, the data collection according to claim 4 is implemented Method steps.