WO2016180228A1 - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- WO2016180228A1 WO2016180228A1 PCT/CN2016/080270 CN2016080270W WO2016180228A1 WO 2016180228 A1 WO2016180228 A1 WO 2016180228A1 CN 2016080270 W CN2016080270 W CN 2016080270W WO 2016180228 A1 WO2016180228 A1 WO 2016180228A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- file
- data processing
- module
- processing method
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
Definitions
- the present application relates to, but is not limited to, the field of electronic technology, and in particular, to a data processing method and apparatus.
- the receiving module is configured to: receive a message
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Disclosed is a data processing method, comprising: receiving a message, and importing configuration parameters from a pre-set file; and according to the imported configuration parameters, extracting corresponding valid data in the message, and generating a data table. By means of the technical solution, the separation of configuration parameters from codes is realized, the setting, modification, addition and deletion of the configuration parameters by a client only require the modification of a pre-set file rather than the re-compilation or modification of client codes, thereby facilitating parameter configuration work.
Description
本申请涉及但不限于电子技术领域，尤其涉及一种数据处理方法及装置。The present application relates to, but is not limited to, the field of electronic technology, and in particular, to a data processing method and apparatus.
网管***通常需要对其管理下的一个或多个网元设备进行各项性能的巡检，用户在网管***的客户端上选择需要巡检的网元设备、巡检项(如温度、风扇转速、功率等)后，网管***的服务端对用户选择的网元设备进行被选的每个巡检项的巡检，并将巡检结果以报文的形式发给客户端，客户端收到报文后，需要根据预先的配置参数提取报文中的有效数据，将每一巡检项的巡检结果生成一个数据表。然而，这种方式有以下几点缺陷：The network management system usually needs to perform the inspection of the performance of one or more NEs under the management. The user selects the NE device and inspection items (such as temperature and fan speed) that need to be inspected on the client of the network management system. After the power is transmitted, the server of the network management system performs the inspection of each selected inspection item on the selected network element device, and sends the inspection result to the client in the form of a packet, and the client receives the packet. After the packet is received, the valid data in the packet is extracted according to the pre-configuration parameters, and the inspection result of each inspection item is generated into a data table. However, this approach has the following drawbacks:
一、客户端对配置参数的设定、修改、增加、删除都不方便，通常需要重新编写客户端代码，因为其配置参数基本都是预先写死在客户端代码中。例如当需要增加巡检项或修改巡检项时，非常繁琐。First, the client is inconvenient to set, modify, add, delete the configuration parameters, usually need to rewrite the client code, because its configuration parameters are basically pre-written in the client code. For example, when it is necessary to add a patrol item or modify a patrol item, it is very cumbersome.
二、针对每一个巡检项，都是分别生成一个数据表，没有自动合并数据表的功能，这导致查看时需要一一打开每一个数据表，不方便。Second, for each inspection item, each data sheet is generated separately, and there is no function of automatically merging the data table, which makes it unnecessary to open each data table one by one when viewing.
三、生成数据表的同时，不能及时地对内存进行释放，导致内存溢出的问题。相关技术中，客户端生成数据表的过程中，客户端用的是Java语言，Java的GC(Garbage Collection，垃圾回收)机制是自动进行的，Java的内存分配和回收也主要在Java的堆上进行，Java的堆中存储了大量的对象实例。从报文中提取了有效数据后，大量的巡检结果存储在堆中，而为每一个巡检项生成Excel文件的过程，就是将巡检结果从堆中取出来放入Excel文件中，由于Excel文件也是存储在堆中，相当于巡检结果在内存中存储了两份，极大地浪费了内存性能。而Java内存自动回收是等到整个Excel文件都生成完毕才会执行。Third, while generating the data table, the memory cannot be released in time, resulting in memory overflow. In the related art, in the process of generating a data table by the client, the client uses the Java language, and the Java GC (Garbage Collection, garbage collection) mechanism is automatically performed. The memory allocation and recovery of Java are also mainly on the heap of Java. In progress, a large number of object instances are stored in the Java heap. After the valid data is extracted from the message, a large number of inspection results are stored in the heap, and the process of generating an Excel file for each inspection item is to take the inspection result from the heap and put it into the Excel file. The Excel file is also stored in the heap, which is equivalent to the inspection result stored in memory two copies, greatly wasting memory performance. The automatic recovery of Java memory is executed until the entire Excel file is generated.
尤其在大规模网元的环境下，经常需要对配置参数进行设定、修改、增加、删除，造成很大工作量；巡检项也特别多，最后会生成大量的数据表，
不能自动合并；生成数据表的过程中内存缓存的数据量非常大，极其容易导致内存溢出。Especially in the environment of large-scale network elements, it is often necessary to set, modify, add, and delete configuration parameters, resulting in a large workload; the inspection items are also particularly large, and finally a large number of data tables are generated.
Can not be automatically merged; the amount of data cached in the process of generating data tables is very large, extremely easy to cause memory overflow.
发明内容Summary of the invention
以下是对本文详细描述的主题的概述。本概述并非是为了限制权利要求的保护范围。The following is an overview of the topics detailed in this document. This Summary is not intended to limit the scope of the claims.
本发明实施例提供一种数据处理方法及装置，解决由于对报文的有效数据进行提取的配置参数与客户端代码绑定，对配置参数的设定、修改、增加、删除需要重新编写或修改代码导致不方便的问题。The embodiment of the invention provides a data processing method and device, which solves the problem that the configuration parameter extracted from the valid data of the packet is bound to the client code, and the setting, modification, addition, and deletion of the configuration parameter need to be rewritten or modified. The code causes inconvenient problems.
本发明实施例采用以下技术方案:The embodiment of the invention adopts the following technical solutions:
一种数据处理方法，包括：A data processing method comprising:
接收报文，并从预设文件中导入配置参数；Receive a message and import configuration parameters from the preset file;
根据导入的所述配置参数提取所述报文中对应的有效数据，并生成数据表。Extracting corresponding valid data in the packet according to the imported configuration parameter, and generating a data table.
可选地，所述报文中包括至少两个预设项目的有效数据；所述生成数据表包括：生成一个数据表文件，所述数据表文件中包括所述至少两个预设项目的有效数据。Optionally, the message includes valid data of at least two preset items; the generating the data table includes: generating a data table file, where the data table file includes valid of the at least two preset items data.
可选地，所述数据表文件为Excel文件，每个预设项目的有效数据分别作为其中的一个sheet页。Optionally, the data table file is an Excel file, and valid data of each preset item is respectively used as one of the sheet pages.
可选地，所述配置参数包括：数据表的列名称或数据表的行名称，以及数据表的值类型。Optionally, the configuration parameter includes: a column name of the data table or a row name of the data table, and a value type of the data table.
可选地，所述数据处理方法还包括：生成所述预设文件；和/或，对所述预设文件中的配置参数进行修改、增加和/或删减。Optionally, the data processing method further includes: generating the preset file; and/or modifying, adding, and/or deleting configuration parameters in the preset file.
可选地，所述生成数据表包括：Optionally, the generated data table includes:
在内存中根据所述配置参数和提取出的有效数据生成数据表格式的表格数据，按照预设方式根据所述表格数据生成数据表并缓存到硬盘中。The table data in the data table format is generated in the memory according to the configuration parameter and the extracted valid data, and the data table is generated according to the table data according to a preset manner and cached into the hard disk.
可选地，在生成表格数据的同时，所述数据处理方法还包括：从内存中
删除生成所述表格数据所对应的有效数据。Optionally, while the table data is generated, the data processing method further includes: from the memory
Delete the valid data corresponding to the generated table data.
可选地，所述按照预设方式根据所述表格数据生成数据表并缓存到硬盘中包括：Optionally, the generating, according to the preset manner, the data table according to the table data and caching to the hard disk includes:
将N行或M列表格数据转化为字节流，N、M为大于或等于1的正整数；Convert N rows or M list cells into byte streams, N, M being positive integers greater than or equal to 1;
将所述字节流转化为文件流并缓存到硬盘中，重复以上步骤，直到所有的表格数据都转化成文件流并且缓存到硬盘中。The byte stream is converted to a file stream and cached to the hard disk, and the above steps are repeated until all the form data is converted into a file stream and cached to the hard disk.
可选地，在将N行或M列表格数据以文件流形式缓存到硬盘中的同时，所述数据处理方法还包括：将内存中的该N行或M列表格数据所转化成的文件流删除。Optionally, while the N rows or M list cells are cached in the file stream to the hard disk, the data processing method further includes: converting the N rows or M list cells in the memory into a file stream. delete.
一种数据处理装置，包括：A data processing device comprising:
接收模块，设置为：接收报文；The receiving module is configured to: receive a message;
导入模块，设置为：从预设文件中导入配置参数；Import the module, set to: import configuration parameters from the preset file;
处理模块，设置为：根据所述导入模块导入的所述配置参数提取所述报文中对应的有效数据，并生成数据表。The processing module is configured to: extract corresponding valid data in the packet according to the configuration parameter imported by the import module, and generate a data table.
可选地，所述处理模块包括：Optionally, the processing module includes:
提取子模块，设置为：根据所述导入模块导入的所述配置参数提取所述报文中对应的有效数据；The extraction sub-module is configured to: extract corresponding valid data in the packet according to the configuration parameter imported by the import module;
第一生成子模块，设置为：在内存中根据所述配置参数和所述提取子模块提取出的有效数据生成数据表格式的表格数据；The first generation submodule is configured to: generate, in the memory, table data in a data table format according to the configuration parameter and the valid data extracted by the extraction submodule;
第二生成子模块，设置为：按照预设方式根据所述第一生成子模块生成的表格数据生成数据表并缓存到硬盘中。The second generation sub-module is configured to: generate a data table according to the table data generated by the first generation sub-module according to a preset manner, and cache the data table into the hard disk.
可选地，所述数据处理装置还包括第一删除子模块，设置为：在所述第一生成子模块生成表格数据的同时，从内存中删除生成所述表格数据所对应的有效数据。Optionally, the data processing apparatus further includes a first deletion submodule, configured to: when the first generation submodule generates the table data, delete the valid data corresponding to the generated table data from the memory.
可选地，所述第二生成子模块是设置为：将N行或M列表格数据转化为字节流；将所述字节流转化为文件流并缓存到硬盘中，重复以上步骤，直到所有的表格数据都转化成文件流并且缓存到硬盘中，其中，N、M为大于或
等于1的正整数。Optionally, the second generation submodule is configured to: convert N rows or M list data into a byte stream; convert the byte stream into a file stream and cache it to a hard disk, and repeat the above steps until All tabular data is converted to a file stream and cached to the hard disk, where N, M are greater than or
A positive integer equal to 1.
可选地，所述数据处理装置还包括第二删除子模块，设置为：在所述第二生成子模块将N行或M列表格数据以文件流形式缓存到硬盘中的同时，将内存中的该N行或M列表格数据所转化成的文件流删除。Optionally, the data processing apparatus further includes a second deletion submodule, configured to: when the second generation submodule caches N rows or M list data in a file stream to the hard disk, The file stream converted by the N-line or M-list data is deleted.
此外，本发明实施例还提供一种计算机可读存储介质，存储有计算机可执行指令，所述计算机可执行指令被执行时实现上述数据处理方法。In addition, an embodiment of the present invention further provides a computer readable storage medium storing computer executable instructions, which are implemented when the computer executable instructions are executed.
本发明实施例采用从预设文件中导入配置数据，当网管***需要对其管理下的一个或多个网元设备进行各项性能的巡检时，用户在网管***的客户端上选择需要巡检的网元设备、巡检项后，网管***的服务端对需要巡检的网元设备进行该每一个巡检项的巡检，并将巡检结果以报文的形式发给客户端，客户端收到报文后，根据导入的配置参数提取报文中的有效数据，生成数据表，由于配置参数与代码实现了分离，客户端对配置参数的设定、修改、增加、删除都无需重新编写客户端代码，只需修改预设文件即可。In the embodiment of the present invention, the configuration data is imported from the preset file. When the network management system needs to perform the performance inspection on one or more network element devices under the management, the user selects the patrol on the client of the network management system. After the network element device and the inspection item are checked, the server of the network management system performs the inspection of each inspection item on the network element device that needs to be inspected, and sends the inspection result to the client in the form of a packet. After receiving the packet, the client extracts the valid data in the packet according to the imported configuration parameters, and generates a data table. Since the configuration parameters are separated from the code, the client does not need to set, modify, add, or delete the configuration parameters. Rewrite the client code, just modify the default file.
可选地，针对每一个巡检项，都分别生成一个sheet页，自动合并sheet页成一个Excel文件，只需要打开该Excel文件，就可以方便查看每个巡检项的巡检结果。Optionally, for each inspection item, a sheet page is separately generated, and the sheet page is automatically merged into an Excel file, and only the Excel file needs to be opened, so that the inspection result of each inspection item can be conveniently viewed.
可选地，生成sheet页、Excel文件的同时，及时地对内存进行释放，避免了内存溢出。Optionally, when the sheet page and the Excel file are generated, the memory is released in time to avoid memory overflow.
本发明实施例尤其适合应用于大规模网元的环境下，当然本发明实施例的应用场景不局限于网管***对其管理下的一个或多个网元设备进行各项性能的巡检。The embodiment of the present invention is particularly applicable to the environment of a large-scale network element. Of course, the application scenario of the embodiment of the present invention is not limited to the performance inspection of one or more network element devices under the management of the network management system.
在阅读并理解了附图和详细描述后，可以明白其他方面。Other aspects will be apparent upon reading and understanding the drawings and detailed description.
附图概述BRIEF abstract
图1为本发明一实施例提供的数据处理方法的流程图；FIG. 1 is a flowchart of a data processing method according to an embodiment of the present invention;
图2为本发明一实施例提供的数据处理装置的示意图。FIG. 2 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention.
在本发明实施例中，配置参数不写死在客户端代码中，而是写入在预设文件里，当需要利用该配置参数提取接收报文中的有效数据时，从预设文件中导入配置参数，这样方便了配置参数的修改、增加、删除等操作，因为无需重写编写代码。In the embodiment of the present invention, the configuration parameter is not written in the client code, but is written in the preset file. When the configuration parameter is used to extract the valid data in the received message, the configuration file is imported from the preset file. Configuration parameters, which facilitate the modification, addition, deletion, etc. of configuration parameters, because there is no need to rewrite the writing code.
下面通过具体实施方式结合附图对本申请作进一步详细说明。The present application will be further described in detail below with reference to the accompanying drawings.
图1为本发明一实施例提供的数据处理方法的流程图，请参考图1，本实施例提供的数据处理方法包括如下流程：FIG. 1 is a flowchart of a data processing method according to an embodiment of the present invention. Referring to FIG. 1, the data processing method provided in this embodiment includes the following process:
步骤S101、接收报文，并从预设文件中导入配置参数。Step S101: Receive a message, and import configuration parameters from the preset file.
以网管***对其管理下的网元设备进行性能巡检这一应用场景为例，当网管***需要对其管理下的某些网元设备进行某些项性能的巡检时，用户在网管***的客户端上选择需要巡检的网元设备、巡检项后，网管***的服务端对用户选择的网元设备进行该每一巡检项的巡检，并将巡检结果以报文的形式发给客户端，客户端接收报文，并从预设文件中导入配置参数。For example, when the NMS performs performance check on certain NE devices under the management of the network management system, the NMS is in the network management system. After selecting the NE device to be inspected and the patrol item, the server of the network management system performs the inspection of each inspection item on the NE device selected by the user, and the inspection result is the packet. The form is sent to the client, and the client receives the message and imports the configuration parameters from the preset file.
可选地，预设文件为XML(Extensible Markup Language，可扩展标记语言)文件。这样当需要新增、修改、删除配置参数的时候，只需要修改XML文件，有利于配置参数的维护，而不需要去代码里修改。Optionally, the preset file is an XML (Extensible Markup Language) file. In this way, when the configuration parameters need to be added, modified, or deleted, only the XML file needs to be modified, which is beneficial to the maintenance of the configuration parameters, and does not need to be modified in the code.
配置参数包括用于从报文中提取有效数据并生成数据表的必要参数，比如可以包括：数据表的列名称或数据表的行名称，以及数据表的值类型；还可以包括数据表的宽度、颜色等等。具体文件的名称，有哪些列，每一列的表头名称，每一列的值怎么显示，如按照网元解析，按照单板解析，按照端口解析，国际化解析，宽度，颜色等，都可以从配置参数中得到相应的指示值，按照这些指示值执行步骤S102。The configuration parameters include necessary parameters for extracting valid data from the message and generating a data table, such as: a column name of the data table or a row name of the data table, and a value type of the data table; and may also include a width of the data table , color, etc. The name of the specific file, which columns, the header name of each column, how to display the value of each column, such as according to the network element analysis, according to the board analysis, according to port analysis, international analysis, width, color, etc., can be Corresponding indication values are obtained in the configuration parameters, and step S102 is performed according to the indication values.
步骤S102、根据导入的所述配置参数提取所述报文中对应的有效数据，并生成数据表。Step S102: Extract corresponding valid data in the packet according to the imported configuration parameter, and generate a data table.
报文中一般包括了配置参数对应的编码后的值，对报文进行解码后把报文的每个属性值取出来，例如利用Java中的反射功能，就可以获取任何属性的名称和值，再根据配置参数从中提取对本次生成数据表有用的有效数据，再根据该配置参数和提取出的有效数据生成数据表。
The packet generally includes the encoded value corresponding to the configuration parameter. After decoding the packet, each attribute value of the packet is extracted. For example, by using the reflection function in Java, the name and value of any attribute can be obtained. Then, according to the configuration parameter, the valid data useful for the generated data table is extracted, and the data table is generated according to the configuration parameter and the extracted valid data.
本实施例，当应用场景中是对两个或两个以上巡检项进行巡检时，那么步骤S101接收的报文中包括至少两个预设项目(比如巡检项)的有效数据；可选地，步骤S102中生成一个数据表文件，该数据表文件中包括所述至少两个预设项目的有效数据，即实现了对多个预设项目的有效数据的合并，合并到一个数据表文件中。可选地，生成的该一个数据表文件为Excel文件，而每个预设项目的有效数据分别作为其中的一个sheet页，这样的话，每个巡检项的巡检结果就是Excel文件当中的一个sheet页，无论巡检项有多少个，用户始终只需要打开一个Excel文件就能浏览所有的巡检结果，能有效改善用户体验。In this embodiment, when the application scenario is to patrol two or more patrol items, the packet received in step S101 includes valid data of at least two preset items (such as patrol items); Optionally, in step S102, a data table file is generated, where the data table file includes valid data of the at least two preset items, that is, the combination of valid data of the plurality of preset items is realized, and the data is merged into one data table. In the file. Optionally, the generated one of the data table files is an Excel file, and the valid data of each preset item is respectively used as one of the sheet pages, so that the inspection result of each inspection item is one of the Excel files. The sheet page, regardless of the number of inspection items, users only need to open an Excel file to browse all the inspection results, which can effectively improve the user experience.
本实施例提供的数据处理方法还可以包括以下步骤：生成步骤S101中的预设文件；和/或，对该预设文件中的配置参数进行修改、增加和/或删减。该步骤相比相关技术中重新编写客户端代码或修改客户端代码的方式，便利许多。The data processing method provided in this embodiment may further include the following steps: generating the preset file in step S101; and/or modifying, adding, and/or deleting the configuration parameter in the preset file. This step is much more convenient than the way in which the client code is rewritten or the client code is modified in the related art.
本实施例中，可以在内存中根据该配置参数和提取出的有效数据生成数据表格式的表格数据，按照预设方式根据该表格数据生成数据表并缓存到硬盘中。为了防止内存溢出，可选地，在生成表格数据的同时，从内存中删除生成该表格数据所对应的有效数据。以生成一个Excel文件，且每个巡检项的巡检结果(从报文中提取的有效数据)分别作为其中一个sheet页为例，在提取出有效数据生成Excel文件的过程中，可以每生成一个sheet页就先从内存中释放掉对应的数据，释放后，内存中就少了一个巡检项的巡检结果数据，而多了一个sheet页，这样相当于在内存中只存储了一份数据，理论上内存消耗减少了一半。In this embodiment, the table data in the data table format may be generated in the memory according to the configuration parameter and the extracted valid data, and the data table is generated according to the table data according to the preset manner and cached into the hard disk. In order to prevent memory overflow, optionally, the valid data corresponding to the generated table data is deleted from the memory while the table data is generated. To generate an Excel file, and the inspection result of each inspection item (effective data extracted from the message) is taken as one of the sheet pages as an example, in the process of extracting valid data to generate an Excel file, each generation can be generated. A sheet page first releases the corresponding data from the memory. After the release, there is one inspection result data of the inspection item in the memory, and an additional sheet page, which is equivalent to storing only one copy in the memory. Data, theoretically, memory consumption has been cut in half.
其中，按照预设方式根据该表格数据生成数据表并缓存到硬盘中这一步骤可以包括：将N行或M列表格数据转化为字节流，N、M为大于或等于1的正整数；将所述字节流转化为文件流并缓存到硬盘中，重复以上步骤，直到所有的表格数据都转化成文件流并且缓存到硬盘中。为了防止内存溢出，可选地，在将N行或M列表格数据以文件流形式缓存到硬盘中的同时，将内存中的该N行或M列表格数据删除。以生成一个Excel文件，且每个巡检项的巡检结果(从报文中提取的有效数据)分别作为其中一个sheet页为例，在
内存中根据配置参数和提取出的有效数据生成Excel格式的表格数据之后，把待生成的Excel文件中的一行行或一列列表格数据转化为字节流，然后将字节流转化为文件流并缓存到硬盘中，这样不需要一次性将所有的数据都导入Excel堆中才生成Excel文件，比如，可以先将1000行表格数据转化为字节流，然后转化成文件流，这样生成了一个只有1000行数据的Excel文件，同时缓存到硬盘中，并在内存中清除这1000行表格数据所转化成的文件流。这样内存中一直只保存了1000行数据，极大减少了生成Excel文件对内存的消耗。The step of generating a data table according to the table data according to the preset manner and buffering the data into the hard disk may include: converting N rows or M list data into a byte stream, where N and M are positive integers greater than or equal to 1; The byte stream is converted to a file stream and cached to the hard disk, and the above steps are repeated until all the form data is converted into a file stream and cached to the hard disk. In order to prevent memory overflow, optionally, the N rows or M list cells in the memory are cached into the hard disk as a file stream, and the N rows or M list cells in the memory are deleted. To generate an Excel file, and the inspection result of each inspection item (effective data extracted from the message) is taken as one of the sheet pages as an example.
After generating the tabular data in the Excel format according to the configuration parameters and the extracted valid data in the memory, the row or row of the list data in the Excel file to be generated is converted into a byte stream, and then the byte stream is converted into a file stream and Cache to the hard disk, so you do not need to import all the data into the Excel heap at one time to generate the Excel file. For example, you can convert 1000 rows of table data into a byte stream and then convert it into a file stream, thus generating a The Excel file of 1000 rows of data is simultaneously cached to the hard disk, and the file stream converted by the 1000 rows of table data is cleared in the memory. In this way, only 1000 rows of data are stored in the memory, which greatly reduces the memory consumption of generating Excel files.
图2为本发明一实施例提供的数据处理装置的示意图，如图2所示，数据处理装置包括：2 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention. As shown in FIG. 2, the data processing apparatus includes:
接收模块21，设置为：接收报文；The receiving module 21 is configured to: receive a message;
导入模块22，设置为：从预设文件中导入配置参数；The import module 22 is configured to: import configuration parameters from a preset file;
处理模块23，设置为：根据导入模块22导入的所述配置参数提取所述报文中对应的有效数据，并生成数据表。The processing module 23 is configured to: extract corresponding valid data in the packet according to the configuration parameter imported by the import module 22, and generate a data table.
可选地，所述报文中包括至少两个预设项目的有效数据；处理模块23是设置为：生成一个数据表文件，其中，所述数据表文件中包括所述至少两个预设项目的有效数据。Optionally, the message includes valid data of at least two preset items; the processing module 23 is configured to: generate a data table file, where the data table file includes the at least two preset items Valid data.
可选地，处理模块23包括：Optionally, the processing module 23 includes:
提取子模块231，设置为：根据导入模块22导入的所述配置参数提取所述报文中对应的有效数据；The extraction sub-module 231 is configured to: extract corresponding valid data in the packet according to the configuration parameter imported by the import module 22;
第一生成子模块232，设置为：在内存中根据所述配置参数和提取子模块231提取出的有效数据生成数据表格式的表格数据；The first generation submodule 232 is configured to: generate, in the memory, table data in a data table format according to the configuration parameter and the valid data extracted by the extraction submodule 231;
第二生成子模块233，设置为：按照预设方式根据第一生成子模块232生成的表格数据生成数据表并缓存到硬盘中。The second generation sub-module 233 is configured to: generate a data table according to the table data generated by the first generation sub-module 232 according to a preset manner and cache the data table into the hard disk.
可选地，数据处理装置还可以包括第一删除子模块24，设置为：在第一生成子模块232生成表格数据的同时，从内存中删除生成所述表格数据所对应的有效数据。Optionally, the data processing apparatus may further include a first deletion sub-module 24, configured to: when the first generation sub-module 232 generates the table data, delete the valid data corresponding to the generated table data from the memory.
可选地，第二生成子模块233是设置为：将N行或M列表格数据转化为
字节流；将所述字节流转化为文件流并缓存到硬盘中，重复以上步骤，直到所有的表格数据都转化成文件流并且缓存到硬盘中，其中，N、M为大于或等于1的正整数。Optionally, the second generation submodule 233 is configured to: convert N rows or M list data into
Byte stream; convert the byte stream into a file stream and cache it to the hard disk, repeat the above steps until all the table data is converted into a file stream and cached to the hard disk, where N, M is greater than or equal to 1 Positive integer.
可选地，数据处理装置还可以包括第二删除子模块25，设置为：在第二生成子模块233将N行或M列表格数据以文件流形式缓存到硬盘中的同时，将内存中的该N行或M列表格数据所转化成的文件流删除。Optionally, the data processing apparatus may further include a second deletion sub-module 25, configured to: when the second generation sub-module 233 caches the N-line or M-list data in a file stream to the hard disk, The file stream converted by the N-line or M-list data is deleted.
本发明实施例实现了配置参数与客户端代码的分离，对配置参数的设定、修改、增加、删除都无需重新编写或修改客户端代码，只需修改预设文件即可，方便了操作。针对每一个巡检项，可以分别生成一个sheet页，自动合并sheet页成一个Excel文件，只需要打开该Excel文件，就可以方便查看每个巡检项的巡检结果。而且，在生成sheet页、Excel文件的同时，还可以及时地对内存进行释放，避免了内存溢出。本发明实施例尤其适合应用在大规模网元的环境下，当然本发明实施例的应用场景不局限于网管***对其管理下的一个或多个网元设备进行各项性能的巡检。The embodiment of the invention realizes the separation of the configuration parameters from the client code, and does not need to rewrite or modify the client code for setting, modifying, adding, and deleting the configuration parameters, and only needs to modify the preset file to facilitate the operation. For each inspection item, a sheet page can be generated separately, and the sheet page is automatically merged into an Excel file, and only the Excel file needs to be opened, so that the inspection result of each inspection item can be conveniently viewed. Moreover, while the sheet page and the Excel file are generated, the memory can be released in time to avoid memory overflow. The embodiment of the present invention is particularly suitable for application in a large-scale network element environment. Of course, the application scenario of the embodiment of the present invention is not limited to the performance inspection of one or more network element devices managed by the network management system.
此外，本发明实施例还提供一种计算机可读存储介质，存储有计算机可执行指令，所述计算机可执行指令被执行时实现上述数据处理方法。In addition, an embodiment of the present invention further provides a computer readable storage medium storing computer executable instructions, which are implemented when the computer executable instructions are executed.
本领域普通技术人员可以理解上述方法中的全部或部分步骤可通过程序来指令相关硬件(例如处理器)完成，所述程序可以存储于计算机可读存储介质中，如只读存储器、磁盘或光盘等。可选地，上述实施例的全部或部分步骤也可以使用一个或多个集成电路来实现。相应地，上述实施例中的各模块/单元可以采用硬件的形式实现，例如通过集成电路来实现其相应功能，也可以采用软件功能模块的形式实现，例如通过处理器执行存储于存储器中的程序/指令来实现其相应功能。本申请不限制于任何特定形式的硬件和软件的结合。One of ordinary skill in the art will appreciate that all or a portion of the above steps may be performed by a program to instruct related hardware, such as a processor, which may be stored in a computer readable storage medium, such as a read only memory, disk or optical disk. Wait. Alternatively, all or part of the steps of the above embodiments may also be implemented using one or more integrated circuits. Correspondingly, each module/unit in the above embodiment may be implemented in the form of hardware, for example, by implementing an integrated circuit to implement its corresponding function, or may be implemented in the form of a software function module, for example, executing a program stored in the memory by a processor. / instruction to achieve its corresponding function. This application is not limited to any specific combination of hardware and software.
以上内容是结合具体的实施方式对本申请所作的进一步详细说明，不能认定本申请的具体实施只局限于这些说明。对于本申请所属技术领域的普通技术人员来说，在不脱离本申请构思的前提下，还可以做出若干简单推演或替换，都应当视为属于本申请的保护范围。
The above content is a further detailed description of the present application in conjunction with the specific embodiments, and the specific implementation of the present application is not limited to the description. It will be apparent to those skilled in the art that the present invention can be made in the form of the present invention without departing from the scope of the present invention.
本发明实施例提供一种数据处理方法及装置，实现了配置参数与客户端代码的分离，对配置参数的设定、修改、增加、删除都无需重新编写或修改客户端代码，只需修改预设文件即可，方便了操作；针对每一个巡检项，可以分别生成一个sheet页，自动合并sheet页成一个Excel文件，只需要打开该Excel文件，就可以方便查看每个巡检项的巡检结果；在生成sheet页、Excel文件的同时，还可以及时地对内存进行释放，避免了内存溢出。
The embodiment of the invention provides a data processing method and device, which realizes separation of configuration parameters from client code, and does not need to rewrite or modify the client code for setting, modifying, adding, or deleting configuration parameters, and only needs to modify the pre- Set the file, it is convenient for operation; for each inspection item, you can generate a sheet page separately, automatically merge the sheet page into an Excel file, just open the Excel file, you can easily view the inspection of each inspection item. Check the results; while generating the sheet page, Excel file, you can also release the memory in a timely manner, to avoid memory overflow.
Claims (15)
- 一种数据处理方法，包括：A data processing method comprising:接收报文，并从预设文件中导入配置参数；Receive a message and import configuration parameters from the preset file;根据导入的所述配置参数提取所述报文中对应的有效数据，并生成数据表。Extracting corresponding valid data in the packet according to the imported configuration parameter, and generating a data table.
- 如权利要求1所述的数据处理方法，其中，所述报文中包括至少两个预设项目的有效数据；所述生成数据表包括：生成一个数据表文件，所述数据表文件中包括所述至少两个预设项目的有效数据。The data processing method according to claim 1, wherein the message includes valid data of at least two preset items; and the generating the data table comprises: generating a data table file, wherein the data table file includes The valid data of at least two preset items.
- 如权利要求2所述的数据处理方法，其中，所述数据表文件为Excel文件，每个预设项目的有效数据分别作为其中的一个sheet页。The data processing method according to claim 2, wherein said data table file is an Excel file, and valid data of each preset item is taken as one of the sheet pages.
- 如权利要求1所述的数据处理方法，其中，所述配置参数包括：数据表的列名称或数据表的行名称，以及数据表的值类型。The data processing method according to claim 1, wherein said configuration parameter comprises: a column name of a data table or a row name of a data table, and a value type of the data table.
- 如权利要求1所述的数据处理方法，所述数据处理方法还包括：生成所述预设文件；和/或，对所述预设文件中的配置参数进行修改、增加和/或删减。The data processing method according to claim 1, wherein the data processing method further comprises: generating the preset file; and/or modifying, adding, and/or deleting configuration parameters in the preset file.
- 如权利要求1至5任一项所述的数据处理方法，其中，所述生成数据表包括：在内存中根据所述配置参数和提取出的有效数据生成数据表格式的表格数据，按照预设方式根据所述表格数据生成数据表并缓存到硬盘中。The data processing method according to any one of claims 1 to 5, wherein the generating the data table comprises: generating, in the memory, table data in a data table format according to the configuration parameter and the extracted valid data, according to the preset The method generates a data table according to the tabular data and caches it to the hard disk.
- 如权利要求6所述的数据处理方法，在生成表格数据的同时，所述数据处理方法还包括：从内存中删除生成所述表格数据所对应的有效数据。The data processing method according to claim 6, wherein the data processing method further comprises: deleting valid data corresponding to the generation of the form data from the memory while generating the form data.
- 如权利要求6所述的数据处理方法，其中，所述按照预设方式根据所述表格数据生成数据表并缓存到硬盘中包括：The data processing method according to claim 6, wherein the generating the data table according to the table data according to the preset manner and caching it to the hard disk comprises:将N行或M列表格数据转化为字节流，其中，N、M为大于或等于1的正整数；将所述字节流转化为文件流并缓存到硬盘中，重复以上步骤，直到所有的表格数据都转化成文件流并且缓存到硬盘中。Convert N rows or M list cells into a byte stream, where N, M are positive integers greater than or equal to 1; convert the byte stream into a file stream and cache it to the hard disk, repeat the above steps until all The tabular data is converted to a file stream and cached to the hard drive.
- 如权利要求8所述的数据处理方法，在将N行或M列表格数据以文件流形式缓存到硬盘中的同时，所述数据处理方法还包括：将内存中的该N 行或M列表格数据所转化成的文件流删除。The data processing method according to claim 8, wherein the data processing method further comprises: storing the N in the memory while the N rows or M list cells are cached in the file stream to the hard disk. The file stream into which the row or M list data is converted is deleted.
- 一种数据处理装置，包括：A data processing device comprising:接收模块，设置为：接收报文；The receiving module is configured to: receive a message;导入模块，设置为：从预设文件中导入配置参数；Import the module, set to: import configuration parameters from the preset file;处理模块，设置为：根据所述导入模块导入的所述配置参数提取所述报文中对应的有效数据，并生成数据表。The processing module is configured to: extract corresponding valid data in the packet according to the configuration parameter imported by the import module, and generate a data table.
- 如权利要求10所述的数据处理装置，其中，所述报文中包括至少两个预设项目的有效数据；所述处理模块是设置为：生成一个数据表文件，其中，所述数据表文件中包括所述至少两个预设项目的有效数据。The data processing apparatus according to claim 10, wherein said message includes valid data of at least two preset items; said processing module is configured to: generate a data table file, wherein said data table file The valid data of the at least two preset items is included.
- 如权利要求10或11所述的数据处理装置，其中，所述处理模块包括：The data processing device of claim 10 or 11, wherein the processing module comprises:提取子模块，设置为：根据所述导入模块导入的所述配置参数提取所述报文中对应的有效数据；The extraction sub-module is configured to: extract corresponding valid data in the packet according to the configuration parameter imported by the import module;第一生成子模块，设置为：在内存中根据所述配置参数和所述提取子模块提取出的有效数据生成数据表格式的表格数据；The first generation submodule is configured to: generate, in the memory, table data in a data table format according to the configuration parameter and the valid data extracted by the extraction submodule;第二生成子模块，设置为：按照预设方式根据所述第一生成子模块生成的表格数据生成数据表并缓存到硬盘中。The second generation sub-module is configured to: generate a data table according to the table data generated by the first generation sub-module according to a preset manner, and cache the data table into the hard disk.
- 如权利要求12所述的数据处理装置，所述数据处理装置还包括第一删除子模块，设置为：在所述第一生成子模块生成表格数据的同时，从内存中删除生成所述表格数据所对应的有效数据。The data processing device of claim 12, wherein the data processing device further comprises a first deletion sub-module, configured to: delete the generated table data from the memory while the first generation sub-module generates the table data Corresponding valid data.
- 如权利要求12所述的数据处理装置，其中，所述第二生成子模块是设置为：将N行或M列表格数据转化为字节流；将所述字节流转化为文件流并缓存到硬盘中，重复以上步骤，直到所有的表格数据都转化成文件流并且缓存到硬盘中；其中，N、M为大于或等于1的正整数。The data processing apparatus according to claim 12, wherein said second generation submodule is configured to: convert N rows or M list cells into a byte stream; convert said byte stream into a file stream and cache To the hard disk, repeat the above steps until all the tabular data is converted into a file stream and cached to the hard disk; where N, M are positive integers greater than or equal to 1.
- 如权利要求14所述的数据处理装置，所述数据处理装置还包括第二删除子模块，设置为：在所述第二生成子模块将N行或M列表格数据以文件流形式缓存到硬盘中的同时，将内存中的该N行或M列表格数据所转化成的文件流删除。 The data processing apparatus according to claim 14, wherein said data processing apparatus further comprises a second deletion sub-module, configured to: cache the N-line or M-list data in a file stream to the hard disk in said second generation sub-module At the same time, the file stream converted into the N line or M list data in the memory is deleted.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510658490.9 | 2015-10-12 | ||
| CN201510658490.9A CN106570037A (en) | 2015-10-12 | 2015-10-12 | Data processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2016180228A1 true WO2016180228A1 (en) | 2016-11-17 |
Family
ID=57248469
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2016/080270 WO2016180228A1 (en) | 2015-10-12 | 2016-04-26 | Data processing method and device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN106570037A (en) |
| WO (1) | WO2016180228A1 (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108874844B (en) * | 2017-11-24 | 2021-05-14 | 视联动力信息技术股份有限公司 | Form data processing method and video network server |
| CN108574609B (en) * | 2017-12-29 | 2021-05-14 | 视联动力信息技术股份有限公司 | Transmission monitoring method and device |
| CN111597596B (en) * | 2020-04-03 | 2023-08-15 | 浙江众合科技股份有限公司 | Automatic construction drawing method and device based on signal system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1567287A (en) * | 2003-06-11 | 2005-01-19 | 中兴通讯股份有限公司 | Centralized broad spectrum report generation method based on expandable sign language |
| CN101140562A (en) * | 2007-02-14 | 2008-03-12 | 中兴通讯股份有限公司 | Method for realizing user-defined report forms |
| CN102760118A (en) * | 2011-04-25 | 2012-10-31 | 中兴通讯股份有限公司 | Method and device for exporting data as Excel file |
| CN104135737A (en) * | 2014-05-20 | 2014-11-05 | 浙江明讯网络技术有限公司 | Method of obtaining network element configuration data of LTE (Long Term Evolution) network base station |
-
2015
- 2015-10-12 CN CN201510658490.9A patent/CN106570037A/en active Pending
-
2016
- 2016-04-26 WO PCT/CN2016/080270 patent/WO2016180228A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1567287A (en) * | 2003-06-11 | 2005-01-19 | 中兴通讯股份有限公司 | Centralized broad spectrum report generation method based on expandable sign language |
| CN101140562A (en) * | 2007-02-14 | 2008-03-12 | 中兴通讯股份有限公司 | Method for realizing user-defined report forms |
| CN102760118A (en) * | 2011-04-25 | 2012-10-31 | 中兴通讯股份有限公司 | Method and device for exporting data as Excel file |
| CN104135737A (en) * | 2014-05-20 | 2014-11-05 | 浙江明讯网络技术有限公司 | Method of obtaining network element configuration data of LTE (Long Term Evolution) network base station |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106570037A (en) | 2017-04-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10437907B2 (en) | Link expansion service | |
| KR101392559B1 (en) | Dynamic media content previews | |
| CN100591078C (en) | A website buffering method and device | |
| CN103731483B (en) | Virtual file system based on cloud computing | |
| CN107231415B (en) | The network storage method and device of picture | |
| CN105404652A (en) | Mass small file processing method based on HDFS | |
| US20160170953A1 (en) | Dom snapshot capture | |
| CN105339932A (en) | Browser-driven power saving | |
| TWI592807B (en) | Method and device for web style address merge | |
| WO2016180228A1 (en) | Data processing method and device | |
| CN104156341A (en) | Online reading system and method | |
| CN103648019A (en) | Video downloading method and device based on HLS protocol | |
| Bhoedjang et al. | Engineering an online computer forensic service | |
| CN107958079A (en) | Aggregate file delet method, system, device and readable storage medium storing program for executing | |
| CN102646135A (en) | Webpage collecting method, device and system | |
| WO2020211367A1 (en) | Data crawling method and apparatus, computer device and storage medium | |
| CN102799372A (en) | Uploading method and uploading device for pictorial information | |
| CN104750741A (en) | Invalid link processing method and invalid link processing device | |
| CN104424242A (en) | Multi-media file processing method and system | |
| CN106372109A (en) | Internet resource file caching method and apparatus | |
| WO2020000812A1 (en) | Method and apparatus for generating ppt document, storage medium, and computer device | |
| EP2856359B1 (en) | Systems and methods for storing data and eliminating redundancy | |
| Yin et al. | Research on scrapy-based distributed crawler system for crawling semi-structure information at high speed | |
| WO2015078255A1 (en) | Video title display method and system, and browser | |
| CN103617224A (en) | Webpage collecting method, webpage collecting device and webpage collecting system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 16792078 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 16792078 Country of ref document: EP Kind code of ref document: A1 |