CN108009071A - The method of Hadoop clusters test - Google Patents
The method of Hadoop clusters test Download PDFInfo
- Publication number
- CN108009071A CN108009071A CN201711270115.2A CN201711270115A CN108009071A CN 108009071 A CN108009071 A CN 108009071A CN 201711270115 A CN201711270115 A CN 201711270115A CN 108009071 A CN108009071 A CN 108009071A
- Authority
- CN
- China
- Prior art keywords
- file
- values
- normal
- hadoop
- distributed memory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012360 testing method Methods 0.000 title claims abstract description 18
- 238000000034 method Methods 0.000 title claims abstract description 16
- 230000000052 comparative effect Effects 0.000 claims abstract description 7
- 238000001514 detection method Methods 0.000 claims abstract description 6
- 230000006870 function Effects 0.000 claims abstract description 5
- 238000012795 verification Methods 0.000 claims abstract description 3
- 230000002159 abnormal effect Effects 0.000 claims description 6
- 238000012217 deletion Methods 0.000 claims description 3
- 230000037430 deletion Effects 0.000 claims description 3
- 230000000694 effects Effects 0.000 description 3
- 239000004744 fabric Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000000712 assembly Effects 0.000 description 1
- 238000000429 assembly Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 230000006641 stabilisation Effects 0.000 description 1
- 238000013112 stability test Methods 0.000 description 1
- 238000011105 stabilization Methods 0.000 description 1
- 238000010998 test method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3034—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a storage system, e.g. DASD based or network based
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3055—Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0805—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability
- H04L43/0817—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability by checking functioning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Abstract
The method of Hadoop clusters test, is written and read distributed memory system different size file operation and file verification, whether detection service device product mix hadoop distributed memory systems function is normal, specifically includes in the following manner：Obtain the md5 values of the file of different capabilities size；Delete the file for having obtained md5 values；Obtain the md5 values of new file；By the new md5 values for obtaining file compared with the md5 values of original, judge whether file is normal according to comparative result；If normal, next step operation is carried out, otherwise stops testing and exporting result；File is saved as more parts；Judge whether the md5 values of more parts of files are consistent, judge whether distributed memory system is normal according to judging result.
Description
Technical field
The present invention relates to distributed memory system technical field of measurement and test, specifically a kind of side of Hadoop clusters test
Method.
Background technology
With the arrival of internet, server has been applied to industry-by-industry field, and the needs of types of applications create not
With the generation of cluster；The epoch server demands quantity of big data shows the trend of straight rise, and all trades and professions are to server
The increase of different application demands and quantity required is generated, to meet product reliability, stability test demand needs, service
Device research and development need to introduce clustered deploy(ment) and using testing scheme.
The content of the invention
It is an object of the invention to provide a kind of method of Hadoop clusters test, for detection service device product mix
Whether hadoop distributed memory systems function is normal.
The technical scheme adopted by the invention to solve the technical problem is that：
The method of Hadoop clusters test, operation and file school are written and read to distributed memory system different size file
Test, whether detection service device product mix hadoop distributed memory systems function is normal, specifically includes in the following manner：
Obtain the md5 values of the file of different capabilities size；
Delete the file for having obtained md5 values；
Obtain the md5 values of new file；
By the new md5 values for obtaining file compared with the md5 values of original, judge file whether just according to comparative result
Often；If normal, next step operation is carried out, otherwise stops testing and exporting result；
File is saved as more parts；
Judge whether the md5 values of more parts of files are consistent, judge whether distributed memory system is normal according to judging result.
Further, needed before the md5 values of file of different capabilities size are obtained below deletion/home/hadoop
The All Files started with file.
Further, obtained md5 values file include/home/hadoop below with md5sum start file.
Further, delete obtained the file of md5 values while, it is also necessary to delete/home below opened with file
The file of head.
Further, the file that also needs to create uploads to hadoop before the md5 values of new file are obtained, and from
Hadoop downloads the file into local/home catalogues.
Further, the new md5 values for obtaining file are judged compared with the md5 values of original according to comparative result
Systemic-function whether be specially normally：
If the md5 values for newly obtaining file are identical with the md5 values of original, then it represents that systemic-function is normal；
If newly obtain the md5 values of file and the md5 values of original differ, then it represents that systemic-function is abnormal.
Further, judge to further include before whether the md5 values of more parts of files are consistent：
Respectively enter more parts of files and obtain the md5 values of local save file and copy to Master hosts /home mesh
Record.
Further, according to judging result judge distributed memory system whether be specially normally：
If the md5 values of more parts of files are consistent, then it represents that distributed memory system is normal, if it is inconsistent, representing to divide
Cloth storage system is abnormal；
Output test result.
What the above content of the invention provided is only the statement of the embodiment of the present invention, rather than invention is in itself.
The effect provided in the content of the invention is only the effect of embodiment, rather than whole effects that invention is all, above-mentioned
A technical solution in technical solution has the following advantages that or beneficial effect：
The problem of can verify that the compatibility of server and distributed memory system platform, realizing compatibility aspect in advance, and
Stability and error rate after server carrying hadoop distributed memory systems, come the overall performance and stabilization of detection service device
Property
Brief description of the drawings
Attached drawing described herein is used for providing that the present invention is explained further, and forms the part of the application, this hair
Bright schematic description and description is used to explain the present invention, does not form inappropriate limitation of the present invention.In the accompanying drawings：
Fig. 1 is the method flow schematic diagram of the embodiment of the present invention.
Embodiment
In order to the technical characterstic of clear explanation this programme, below by embodiment, and its attached drawing is combined, to this
Invention is described in detail.Following disclosure provides many different embodiments or example is used for realizing the different knots of the present invention
Structure.In order to simplify disclosure of the invention, hereinafter the component and setting of specific examples are described.In addition, the present invention can be with
Repeat reference numerals and/or letter in different examples.This repetition is that for purposes of simplicity and clarity, itself is not indicated
Relation between various embodiments are discussed and/or are set.It should be noted that illustrated component is not necessarily to scale in the accompanying drawings
Draw.Present invention omits the description to known assemblies and treatment technology and process to avoid the present invention is unnecessarily limiting.
As shown in Figure 1, the method for Hadoop clusters test, behaviour is written and read to distributed memory system different size file
Work and file verification, whether detection service device product mix hadoop distributed memory systems function is normal, specifically includes following
Mode：
Obtain the md5 values of the file of different capabilities size；
Delete the file for having obtained md5 values；
Obtain the md5 values of new file；
By the new md5 values for obtaining file compared with the md5 values of original, judge file whether just according to comparative result
Often；If normal, next step operation is carried out, otherwise stops testing and exporting result；
File is saved as more parts；
Judge whether the md5 values of more parts of files are consistent, judge whether distributed memory system is normal according to judging result.
In obtain different capabilities size file md5 values before need to open with file below deletion/home/hadoop
The All Files of head.Such as：rm-rf/home/hadoop/file*.
Obtained md5 values file include/home/hadoop below with md5sum start file, file- $
imd5is$md5sum$i。
Delete obtained the file of md5 values while, it is also necessary to delete/home below with file beginning file.
There is Stub File to influence test result before preventing.Such as：
rm-rf/home/file*
rm–rf/home/hadoop/md5sum*
The file for also needing to create before the md5 values of new file are obtained uploads to hadoop, and under hadoop
Part publish papers to local/home catalogues.
By the new md5 values for obtaining file compared with the md5 values of original, judge that systemic-function is according to comparative result
It is no to be specially normally：
If the md5 values for newly obtaining file are identical with the md5 values of original, then it represents that systemic-function is normal；
If newly obtain the md5 values of file and the md5 values of original differ, then it represents that systemic-function is abnormal.
Judge to further include before whether the md5 values of more parts of files are consistent：
Respectively enter more parts of files and obtain the md5 values of local save file and copy to Master hosts /home mesh
Record.
According to judging result judge distributed memory system whether be specially normally：
If the md5 values of more parts of files are consistent, then it represents that distributed memory system is normal, if it is inconsistent, representing to divide
Cloth storage system is abnormal；
Output test result.
The above is the preferred embodiment of the present invention, for those skilled in the art,
Without departing from the principles of the invention, some improvements and modifications can also be made, these improvements and modifications are also regarded as this hair
Bright protection domain.
Claims (8)
- The method of 1.Hadoop clusters test, it is characterized in that, operation is written and read to distributed memory system different size file And file verification, whether detection service device product mix hadoop distributed memory systems function is normal, specifically includes with lower section Formula：Obtain the md5 values of the file of different capabilities size；Delete the file for having obtained md5 values；Obtain the md5 values of new file；By the new md5 values for obtaining file compared with the md5 values of original, judge whether file is normal according to comparative result； If normal, next step operation is carried out, otherwise stops testing and exporting result；File is saved as more parts；Judge whether the md5 values of more parts of files are consistent, judge whether distributed memory system is normal according to judging result.
- 2. according to the method described in claim 1, it is characterized in that, obtain different capabilities size file md5 values before need Delete/home/hadoop below with file start All Files.
- 3. according to the method described in claim 1, it is characterized in that, obtained md5 values file include/home/hadoop under The file started with md5sum in face.
- 4. according to the method described in claim 3, it is characterized in that, delete obtained the file of md5 values while, it is also necessary to The file started below deletion/home with file.
- 5. according to the method described in claim 1, it is characterized in that, obtain new file md5 values before also need to create File upload to hadoop, and download the file into local/home catalogues from hadoop.
- 6. according to the method described in claim 1, it is characterized in that, will the new md5 values for obtaining file and original the progress of md5 values Compare, according to comparative result judge systemic-function whether be specially normally：If the md5 values for newly obtaining file are identical with the md5 values of original, then it represents that systemic-function is normal；If newly obtain the md5 values of file and the md5 values of original differ, then it represents that systemic-function is abnormal.
- 7. according to the method described in claim 1, it is characterized in that, judge to further include before whether the md5 values of more parts of files consistent：Respectively enter more parts of files and obtain the md5 values of local save file and copy to Master hosts /home catalogues.
- 8. according to the method described in claim 7, it is characterized in that, judge whether distributed memory system normal according to judging result Specially：If the md5 values of more parts of files are consistent, then it represents that distributed memory system is normal, if it is inconsistent, representing distributed Storage system is abnormal；Output test result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711270115.2A CN108009071B (en) | 2017-12-05 | 2017-12-05 | Hadoop cluster test method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711270115.2A CN108009071B (en) | 2017-12-05 | 2017-12-05 | Hadoop cluster test method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108009071A true CN108009071A (en) | 2018-05-08 |
| CN108009071B CN108009071B (en) | 2020-10-20 |
Family
ID=62056912
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711270115.2A Active CN108009071B (en) | 2017-12-05 | 2017-12-05 | Hadoop cluster test method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108009071B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109976957A (en) * | 2019-02-11 | 2019-07-05 | 平安科技（深圳）有限公司 | Performance test methods, device and the computer equipment of librgw |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101272275A (en) * | 2008-05-07 | 2008-09-24 | 华为技术有限公司 | Network management equipment, method and system for acquiring network appliance configuration information |
| CN104298933A (en) * | 2014-10-17 | 2015-01-21 | 浪潮(北京)电子信息产业有限公司 | Configuration information safety processing method and system |
| CN105975865A (en) * | 2016-05-27 | 2016-09-28 | 北京工业大学 | Hadoop platform measuring method based on dependable computing |
| CN106059825A (en) * | 2016-07-08 | 2016-10-26 | 福建天晴数码有限公司 | Distributed system and configuration method |
| CN107315668A (en) * | 2017-06-26 | 2017-11-03 | 郑州云海信息技术有限公司 | Distributed memory system data consistency automates quick determination method and device |
-
2017
- 2017-12-05 CN CN201711270115.2A patent/CN108009071B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101272275A (en) * | 2008-05-07 | 2008-09-24 | 华为技术有限公司 | Network management equipment, method and system for acquiring network appliance configuration information |
| CN104298933A (en) * | 2014-10-17 | 2015-01-21 | 浪潮(北京)电子信息产业有限公司 | Configuration information safety processing method and system |
| CN105975865A (en) * | 2016-05-27 | 2016-09-28 | 北京工业大学 | Hadoop platform measuring method based on dependable computing |
| CN106059825A (en) * | 2016-07-08 | 2016-10-26 | 福建天晴数码有限公司 | Distributed system and configuration method |
| CN107315668A (en) * | 2017-06-26 | 2017-11-03 | 郑州云海信息技术有限公司 | Distributed memory system data consistency automates quick determination method and device |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109976957A (en) * | 2019-02-11 | 2019-07-05 | 平安科技（深圳）有限公司 | Performance test methods, device and the computer equipment of librgw |
| CN109976957B (en) * | 2019-02-11 | 2022-04-15 | 平安科技（深圳）有限公司 | Librgw performance test method and device and computer equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108009071B (en) | 2020-10-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104615466B (en) | A kind of cloud platform dispositions method and system | |
| US8713270B2 (en) | Efficient file system metadata scanning using scoped snapshots | |
| WO2020000806A1 (en) | Configuration file comparison method and configuration file comparison device | |
| US20160239409A1 (en) | Testing a web service using inherited test attributes | |
| WO2020164263A1 (en) | Librgw performance test method and apparatus, and computer device | |
| US7359819B2 (en) | Evaluation of device driver output | |
| CN104809042A (en) | Iozone-based hard disk performance testing method | |
| US20150261766A1 (en) | Method and apparatus for determining a range of files to be migrated | |
| US8024707B2 (en) | Facilitating self-remediation for software applications | |
| DE102017104073A1 (en) | Generalized verification procedure for write operations | |
| CN104063303B (en) | A method of obtaining and discharge root authority | |
| WO2019100690A1 (en) | Electronic device, testing method, system and computer readable storage medium | |
| CN108009071A (en) | The method of Hadoop clusters test | |
| CA2811617A1 (en) | Commit sensitive tests | |
| CN110502242A (en) | Code automatic generation method, device, computer equipment and storage medium | |
| CN110888648A (en) | Method for generating custom linux image through interactive interface | |
| US8762781B2 (en) | Method and apparatus useful in manufacturing test case operations | |
| CN111026656A (en) | Automatic testing system, method, equipment and storage medium for distributed storage | |
| US9720919B1 (en) | System and method of assessing the state of a database product for installation consistency | |
| US10956294B2 (en) | Methods and systems for testing storage devices via a representative I/O generator | |
| CN111124772A (en) | Cloud platform storage performance testing method, system, terminal and storage medium | |
| CN107360063A (en) | A kind of automated network method for testing pressure | |
| CN108958727A (en) | A kind of generation method and system of API client code | |
| CN111090583B (en) | Method and device for generating code coverage rate report and computer equipment | |
| CN108346453B (en) | Flash memory test equipment and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200917 Address after: 215100 No. 1 Guanpu Road, Guoxiang Street, Wuzhong Economic Development Zone, Suzhou City, Jiangsu Province Applicant after: SUZHOU LANGCHAO INTELLIGENT TECHNOLOGY Co.,Ltd. Address before: 450018 Henan province Zheng Dong New District of Zhengzhou City Xinyi Road No. 278 16 floor room 1601 Applicant before: ZHENGZHOU YUNHAI INFORMATION TECHNOLOGY Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |