CN107066281B - Method for rapidly loading large data group based on DSS data system - Google Patents
Method for rapidly loading large data group based on DSS data system Download PDFInfo
- Publication number
- CN107066281B CN107066281B CN201710352181.8A CN201710352181A CN107066281B CN 107066281 B CN107066281 B CN 107066281B CN 201710352181 A CN201710352181 A CN 201710352181A CN 107066281 B CN107066281 B CN 107066281B
- Authority
- CN
- China
- Prior art keywords
- data
- server
- memory
- space
- function
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/20—Software design
- G06F8/22—Procedural
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method for rapidly loading a large data group based on a DSS data system, which fully utilizes the hardware resources of a server by opening up a large virtual memory and submitting a large amount of physical memories to the server for use, so that the large data group can be loaded possibly by using the virtual memory, and the memory opening efficiency is obviously improved; the data is convenient and fast to access; the invention also prevents the memory page exchange of the server process space in the operation period to the maximum extent by locking the physical memory, thereby greatly reducing the performance loss of the server; the invention also realizes the one-time loading of the large data group by using the virtual memory, avoids the condition that the data needs to be loaded again in the existing scheme, and particularly has the action of causing downtime on the server for the data loading condition needing the I/O process.
Description
Technical Field
The invention relates to a method for quickly loading a large data group based on a DSS data system.
Background
The Darwin Streaming Server is called DSS for short. The DSS is an open source real-time streaming media playing server program provided by Apple company, the whole program is written by pure C + +, the design follows the program design principles of high performance, simplicity, modularization and the like, the program is required to be efficient, the expandability is good, and therefore, the DSS server system is quite ideal in cross-platform support, can run on Windows NT, Windows 2000 and above Windows kernel versions, and can also run on NIX versions well, including Mac OS X, Linux, FreeBSD and Solaris, and the DSS is developed based on standard streaming media protocols RTSP and RTP/RTCP, so that the application has extremely high universality and universality.
The prior DSS-based server data system has the following defects:
firstly, a chimney-type architecture on the traditional IT service deployment leads to high allocation and low usage of equipment hardware, in order to meet the performance and capacity bearing requirements of application programs by considering future application system service development and service burst requirements, a user often selects a certain proportion of allowance to be left when configuring hardware equipment such as computation, storage and network, but after hardware resources are on line, the load of the application system in a certain time is not too high, so that the utilization rate of the higher configured hardware equipment is not high.
Secondly, the existing DSS server data system stores data and exchanges data with a heap as a main means, in this case, if the data volume is small, the influence is not great under the condition of infrequent data use, however, for an enterprise business-class server, it is often necessary to load large-batch data and frequently operate large-scale data groups, if an original heap operation mode is used, space is opened and released each time, and the opened space is very large, and a large amount of memory fragments are generated in the operation process, so that the space and time efficiency in memory use are very low, and the program will deteriorate after running for a period of time.
Thirdly, the existing DSS server does not lock a memory page, the server often has a very large memory capacity and is exclusively used, the actual physical memory usage in the actual operation process is very low, page-missing interrupts are frequently generated when a large data cluster is operated, the generation of the page-missing interrupts seriously affects the response speed of the server and further accumulates data access, and finally even causes the server to be down, which is often difficult to solve.
The existing DSS-based framework is usually used for reading and writing a large amount of data from a database in a small batch and in a loading mode when needed, the use limitation of the mode is very large, and the frequent I/O operation is unacceptable consumption in the use of a server.
In view of the above, the present invention provides a method for fast loading a large data group based on a DSS data system.
Disclosure of Invention
The invention aims to provide a method for quickly loading a large data group based on a DSS data system, which is used for fully utilizing physical memory resources of a server, realizing quick loading and high efficiency and practicability of the large data group and improving the efficiency of the DSS frame data system.
In order to achieve the purpose, the technical scheme of the invention is as follows:
a method for rapidly loading large data groups based on a DSS data system comprises the following steps:
step 1, designing a special virtual memory class QTSSVMarray, wherein the virtual memory class is a template class and comprises a construction function for reserving a preset amount of space in initialization, and the space is not in a physical memory;
step 2, taking a QTSSVMArray of a virtual memory template as a static member variable of a QTSSDictionary in the DSS original data system, and designing the type of a structural body to be stored in a storage space according to the requirement of an application program;
step 3, initializing a static member variable QTSSVMArray in QTSSDictionary, reserving 10MB address space, and restarting the DSS data server if the DSS server fails to reserve the space at the stage;
step 4, adding a function SetBigValue and a qtssAttrDataTypeBigData mark in the QTSSDictionary, wherein the function SetBigValue is used for setting and storing a large-scale data structure, the qtssAttrDataTypeBigData mark is used for setting and storing a large-scale structure data type in the system data type, actually submitting the function SetBigValue to a physical memory when the function SetBigValue is called for the first time, and submitting 1MB of an actual physical memory to a previously reserved address space according to the requirement; the SetBliguValue function uses pointers to store data, wherein the length of fAllocatedLen in a qtssAttrDataTypeBigData mark is set as the length of a pointer array in the function, fAttributeData.Ptr is the address of the pointer array, and fAttributeData.Len is the value of a pointer sizeof;
and 5, the server starts a process, a huge data group loaded in a database by a database plug-in enters a server process memory space, a SetBigValue function is designed in the database plug-in and called once for each row of data, each row of data is stored in each structural body in the virtual memory, at the moment, the SetBigValue function corresponds to an actual physical memory structure according to the array subscript in the developed pointer array, and each pointer in the pointer array is respectively and correspondingly submitted to a memory area in the virtual memory physical space for data storage.
Step 6, adding a condition for acquiring the attribute value of the newly added qtssAttrDataTypeBigData in a GetValuePtr function of the DSS data system, wherein the acquisition mode of the address of the newly added virtual memory physical space is the same as the acquisition mode of the physical space using the heap space;
and 7, locking the submitted physical memory in the using process of the server.
Step 8, adding a QTSS _ SetBugValue interface for providing an external plug-in for a server to use;
and 9, fictitious functions in the QTSSVMArray class so as to release the virtual memory life cycle at the end.
The step 1 further comprises: setting MEM _ TOP _ DOWN flag to virtual function VirtualAlloc to make the operating system reserve space from the high memory address.
In step 2, the structure type stored in the storage space may be designed as a structure type with a size of 1024Bytes per structure according to actual needs, and the structure type is designed as a multiple of the size of the target system paging file, which is 4KB on X86 and X64 and 8KB on IA 64.
After adopting the scheme, compared with the prior art, the invention has the beneficial effects that:
(1) by opening up large virtual memory and submitting a large amount of physical memory to the server for use, the hardware resources of the server are fully utilized, and the existing method is idle in a large amount on the use of the server resources.
(2) Through the use of virtual memory, the characteristics of the virtual memory can be fully used:
1) loading of large data clusters becomes possible;
2) the memory is developed at one time, and compared with the existing system which needs to be developed each time, the efficiency is obviously improved;
3) through the characteristic of continuous virtual memory space, the data access is convenient and fast, the address is discontinuous in the physical memory space in the stack-up mode of the existing scheme, the linked list type access is inefficient, and a large amount of memory fragments are easily caused.
(3) The method has the advantages that the physical memory is locked to prevent the memory page exchange of the server process space in the operation period to the maximum extent, so that the performance loss of the server is greatly reduced, and the existing method frequently carries out page swap-in and swap-out scheduling by the memory manager of the operating system in the use process of a large data group, so that the performance of the server is greatly influenced;
(4) the large data cluster is loaded at one time through the use of the virtual memory, the condition that reloading is needed in the existing scheme is avoided, and particularly the data loading condition needing the I/O process is the behavior of downtime on a server.
The invention is further described with reference to the following figures and detailed description.
Drawings
FIG. 1: the invention relates to a flow chart of a design method of a DSS efficient data system.
Detailed Description
As shown in fig. 1, a method for fast loading a large data group based on a DSS data system disclosed in the present invention includes the following steps:
step 1, designing a special virtual memory class QTSSVMarray, wherein the virtual memory class is a template class and comprises a construction function for reserving a set amount of space in advance during initialization, and the space is not in a physical memory; setting an MEM _ TOP _ DOWN mark in a virtual function VirtualAlloc to enable an operating system to reserve a space from a high-order memory address;
step 2, taking a QTSSVMArray of a virtual memory template as a static member variable of a QTSSDictionary in the DSS original data system, and designing the type of a structural body to be stored in a storage space according to the requirement of an application program; the structure type stored in the storage space can be designed into a structure type with the size of 1024Bytes in each structure according to actual needs, and the structure type is designed into a multiple of the size of a target system paging file, and the value is 4KB on X86 and X64 and 8KB on IA 64;
step 3, initializing a static member variable QTSSVMArray in QTSSDictionary, reserving 10MB address space, and restarting the DSS data server if the DSS server fails to reserve the space at the stage;
step 4, adding a function SetBigValue and a qtssAttrDataTypeBigData mark in the QTSSDictionary, wherein the function SetBigValue is used for setting and storing a large-scale data structure, the qtssAttrDataTypeBigData mark is used for setting and storing a large-scale structure data type in the system data type, actually submitting the function SetBigValue to a physical storage when the function SetBigValue is called for the first time, and submitting 1MB of actual physical memory to a previously reserved address space according to the requirement in the text; the SetBliguValue function uses pointers to store data, wherein the length of fAllocatedLen in a qtssAttrDataTypeBigData mark is set as the length of a pointer array in the function, fAttributeData.Ptr is the address of the pointer array, and fAttributeData.Len is the value of a pointer sizeof;
step 5, the server starts the process and enters the memory space of the server process by loading the huge data group in the database through the database plug-in, calling a SetBigValue function once for each row of data in the database plug-in, storing each row of data in each structural body in the virtual memory, wherein the SetBigValue function corresponds to an actual physical memory structure according to the array subscript in the developed pointer array, and each pointer in the pointer array is respectively and correspondingly submitted to a memory area in the physical space of the virtual memory for data storage; in this step, allocation is performed according to the size and scale of the data in the database read by the actual program.
Step 6, adding a condition for acquiring the attribute value of the newly added qtssAttrDataTypeBigData in a GetValuePtr function of the DSS data system, wherein the acquisition mode of the address of the newly added virtual memory physical space is the same as the acquisition mode of the heap space;
and 7, locking the submitted physical memory in the using process of the server to prevent paging exchange.
Step 8, adding a QTSS _ SetBugValue interface for providing an external plug-in for a server to use;
and 9, fictitious functions in the QTSSVMArray class so as to release the virtual memory life cycle at the end.
The above are merely specific examples of the present invention, and do not limit the scope of the present invention. All equivalent changes made according to the design idea of the present application fall within the protection scope of the present application.
Claims (3)
1. A method for rapidly loading large data groups based on a DSS data system is characterized by comprising the following steps:
step 1, designing a special virtual memory class QTSSVMarray, wherein the virtual memory class is a template class and comprises a construction function for reserving a preset amount of space in initialization, and the space is not in a physical memory;
step 2, taking a QTSSVMArray of a virtual memory template as a static member variable of a QTSSDictionary in the DSS original data system, and designing the type of a structural body to be stored in a storage space according to the requirement of an application program;
step 3, initializing a static member variable QTSSVMArray in QTSSDictionary, reserving 10MB address space, and restarting the DSS data server if the DSS server fails to reserve the space at the stage;
step 4, adding a function SetBigValue and a qtssAttrDataTypeBigData mark in the QTSSDictionary, wherein the function SetBigValue is used for setting and storing a large-scale data structure, the qtssAttrDataTypeBigData mark is used for setting and storing a large-scale structure data type in the system data type, actually submitting the function SetBigValue to a physical memory when the function SetBigValue is called for the first time, and submitting 1MB of an actual physical memory to a previously reserved address space according to the requirement; the SetBliguValue function uses pointers to store data, wherein the length of fAllocatedLen in a qtssAttrDataTypeBigData mark is set as the length of a pointer array in the function, fAttributeData.Ptr is the address of the pointer array, and fAttributeData.Len is the value of a pointer sizeof;
step 5, the server starts the process and enters the memory space of the server process by loading the huge data group in the database through the database plug-in, calling a SetBigValue function once for each row of data in the database plug-in, storing each row of data in each structural body in the virtual memory, wherein the SetBigValue function corresponds to an actual physical memory structure according to the array subscript in the developed pointer array, and each pointer in the pointer array is respectively and correspondingly submitted to a memory area in the physical space of the virtual memory for data storage;
step 6, adding a condition for acquiring the attribute value of the newly added qtssAttrDataTypeBigData in a GetValuePtr function of the DSS data system, wherein the acquisition mode of the address of the newly added virtual memory physical space is the same as the acquisition mode of the used heap space;
step 7, locking the submitted physical memory in the using process of the server;
step 8, adding a QTSS _ SetBugValue interface for providing an external plug-in for a server to use;
and 9, fictitious functions in the QTSSVMArray class so as to release the virtual memory life cycle at the end.
2. The method for fast loading of large data groups based on DSS data system according to claim 1, wherein: the step 1 further comprises: setting MEM _ TOP _ DOWN flag to virtual function VirtualAlloc to make the operating system reserve space from the high memory address.
3. The method for fast loading of large data groups based on DSS data system according to claim 1, wherein: in the step 2, the structure type stored in the storage space can be designed into a structure type with a size of 1024Bytes for each structure, and the structure type is designed to be a multiple of the size of the paging file of the target system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710352181.8A CN107066281B (en) | 2017-05-18 | 2017-05-18 | Method for rapidly loading large data group based on DSS data system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710352181.8A CN107066281B (en) | 2017-05-18 | 2017-05-18 | Method for rapidly loading large data group based on DSS data system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107066281A CN107066281A (en) | 2017-08-18 |
| CN107066281B true CN107066281B (en) | 2020-08-11 |
Family
ID=59611099
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710352181.8A Active CN107066281B (en) | 2017-05-18 | 2017-05-18 | Method for rapidly loading large data group based on DSS data system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107066281B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107959672B (en) * | 2017-11-09 | 2020-09-01 | 厦门普杰信息科技有限公司 | Transfer multicast method based on Darwin streaming media server |
| CN108063918B (en) * | 2017-12-14 | 2020-02-21 | 厦门普杰信息科技有限公司 | Method for gathering and forwarding GB28181 media stream |
| CN108958926B (en) * | 2018-05-25 | 2021-09-07 | 厦门普杰信息科技有限公司 | Darwin streaming media server-based virtual memory pool design method |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103150257A (en) * | 2013-02-28 | 2013-06-12 | 天脉聚源(北京)传媒科技有限公司 | Memory management method and memory management device |
| CN103853665A (en) * | 2012-12-03 | 2014-06-11 | 华为技术有限公司 | Storage space allocation method and device |
| US9430400B2 (en) * | 2013-03-14 | 2016-08-30 | Nvidia Corporation | Migration directives in a unified virtual memory system architecture |
-
2017
- 2017-05-18 CN CN201710352181.8A patent/CN107066281B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103853665A (en) * | 2012-12-03 | 2014-06-11 | 华为技术有限公司 | Storage space allocation method and device |
| CN103150257A (en) * | 2013-02-28 | 2013-06-12 | 天脉聚源(北京)传媒科技有限公司 | Memory management method and memory management device |
| US9430400B2 (en) * | 2013-03-14 | 2016-08-30 | Nvidia Corporation | Migration directives in a unified virtual memory system architecture |
Non-Patent Citations (1)
| Title |
|---|
| 基于高清视频的达尔文流媒体服务器研究和性能改良;曹光利;《重庆邮电大学硕士学位论文》;20131230;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107066281A (en) | 2017-08-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8135904B2 (en) | Method and apparatus for facilitating fast wake-up of a non-volatile memory system | |
| CN106156278B (en) | Database data reading and writing method and device | |
| CN107066281B (en) | Method for rapidly loading large data group based on DSS data system | |
| CN110750372B (en) | Log system and log management method based on shared memory | |
| CN103019955B (en) | The EMS memory management process of PCR-based AM main memory application | |
| CN103150257A (en) | Memory management method and memory management device | |
| WO2011048268A1 (en) | Saving snapshot of a knowledge base without blocking | |
| CN103810429A (en) | Computer virus searching and killing method based on desktop cloud virtualization technology | |
| WO2021169496A1 (en) | Data reading method, apparatus, and device, and storage medium | |
| CN114416425A (en) | Process copying method and device | |
| CN103927215A (en) | kvm virtual machine scheduling optimization method and system based on memory disk and SSD disk | |
| CN102209016B (en) | Data processing method, device and data processing system | |
| CN108932112B (en) | Data read-write method, device, equipment and medium for solid particles | |
| CN105468400A (en) | Linux user mode based method and system for calling timer | |
| US20110264903A1 (en) | Electronic device and method for loading configuration files using the same | |
| CN103020077A (en) | Method for managing memory of real-time database of power system | |
| CN101799832A (en) | Method for rapidly writing mass data into database once | |
| CN109507991A (en) | A kind of two axis servo control platform debugging system and method | |
| CN105183854A (en) | Scheduling method applicable to data unloading of database | |
| CN113986775B (en) | Page table item generation method, system and device in RISC-V CPU verification | |
| CN105162684A (en) | Group communication system and method | |
| US8886675B2 (en) | Method and system for managing data clusters | |
| CN115221076A (en) | High-performance key value storage method and system based on persistent CPU cache | |
| CN109947365B (en) | Distributed storage data verification method and device | |
| CN106970826A (en) | A kind of page fault solution based on big page |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address | ||
| CP03 | Change of name, title or address |
Address after: 4 / F, 281 Lianqian West Road, Siming District, Xiamen City, Fujian Province 361007 Patentee after: Xiamen Municipal Smart City Technology Co.,Ltd. Address before: Siming District of Xiamen City, Fujian province 361000 lianqianxi No. 281 4 floor Patentee before: XIAMEN PUJIE INFORMATION TECHNOLOGY Co.,Ltd. |
|
| CP02 | Change in the address of a patent holder | ||
| CP02 | Change in the address of a patent holder |
Address after: 361000 13th floor, municipal building, No. 2777, Yunding Middle Road, Huli District, Xiamen City, Fujian Province Patentee after: Xiamen Municipal Smart City Technology Co.,Ltd. Address before: 361007 4th floor, 281 Lianqian West Road, Siming District, Xiamen City, Fujian Province Patentee before: Xiamen Municipal Smart City Technology Co.,Ltd. |