CN108009071B - Hadoop cluster test method - Google Patents
Hadoop cluster test method Download PDFInfo
- Publication number
- CN108009071B CN108009071B CN201711270115.2A CN201711270115A CN108009071B CN 108009071 B CN108009071 B CN 108009071B CN 201711270115 A CN201711270115 A CN 201711270115A CN 108009071 B CN108009071 B CN 108009071B
- Authority
- CN
- China
- Prior art keywords
- file
- value
- files
- normal
- values
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000010998 test method Methods 0.000 title description 2
- 238000012360 testing method Methods 0.000 claims abstract description 16
- 238000000034 method Methods 0.000 claims abstract description 14
- 238000012795 verification Methods 0.000 claims abstract description 4
- 230000002159 abnormal effect Effects 0.000 claims description 3
- 238000012217 deletion Methods 0.000 claims 1
- 230000037430 deletion Effects 0.000 claims 1
- 230000000694 effects Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 230000000630 rising effect Effects 0.000 description 1
- 238000013112 stability test Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3034—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a storage system, e.g. DASD based or network based
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3055—Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0805—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability
- H04L43/0817—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability by checking functioning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Abstract
The Hadoop cluster testing method comprises the following steps of performing read-write operation and file verification on files with different sizes of a distributed storage system, and detecting whether functions of a server product matched with the Hadoop distributed storage system are normal or not, wherein the method specifically comprises the following steps: acquiring md5 values of files with different capacities; deleting the file which has acquired the md5 value; acquiring the md5 value of the new file; comparing the md5 value of the newly acquired file with the md5 value of the original file, and judging whether the file is normal or not according to the comparison result; if the result is normal, performing the next operation, otherwise, stopping the test and outputting the result; saving the file as a plurality of copies; and judging whether the md5 values of the files are consistent or not, and judging whether the distributed storage system is normal or not according to the judgment result.
Description
Technical Field
The invention relates to the technical field of distributed storage system testing, in particular to a Hadoop cluster testing method.
Background
With the arrival of the internet, the server has been applied to various industry fields, and the requirements of various applications create different clusters; the demand quantity of servers in the era of big data shows a straight-line rising trend, different application demands and the increase of the demand quantity are generated on the servers in various industries, and cluster deployment and application test schemes are required to be introduced for the research and development of the servers to meet the requirements of the reliability and stability test of products.
Disclosure of Invention
The invention aims to provide a Hadoop cluster testing method which is used for detecting whether a server product is matched with a Hadoop distributed storage system to have normal functions.
The technical scheme adopted by the invention for solving the technical problems is as follows:
the Hadoop cluster testing method comprises the following steps of performing read-write operation and file verification on files with different sizes of a distributed storage system, and detecting whether functions of a server product matched with the Hadoop distributed storage system are normal or not, wherein the method specifically comprises the following steps:
acquiring md5 values of files with different capacities;
deleting the file which has acquired the md5 value;
acquiring the md5 value of the new file;
comparing the md5 value of the newly acquired file with the md5 value of the original file, and judging whether the file is normal or not according to the comparison result; if the result is normal, performing the next operation, otherwise, stopping the test and outputting the result;
saving the file as a plurality of copies;
and judging whether the md5 values of the files are consistent or not, and judging whether the distributed storage system is normal or not according to the judgment result.
Further, all files under/home/hadoop beginning with file need to be deleted before the md5 values for files of different sizes are obtained.
Further, the files for which md5 values have been obtained include files under/home/hadoop beginning with md5 sum.
Further, while deleting the file for which the md5 value has been acquired, it is also necessary to delete the file that begins with the file under the/home.
Further, the created file needs to be uploaded to the hadoop and downloaded from the hadoop to the local/home directory before the md5 value of the new file is obtained.
Further, comparing the md5 value of the newly acquired file with the md5 value of the original file, and judging whether the system functions are normal according to the comparison result:
if the md5 value of the newly acquired file is the same as the md5 value of the original file, the system functions normally;
if the md5 value of the newly acquired file is not the same as the md5 value of the original file, it indicates that the system is not functioning properly.
Further, before determining whether the md5 values of the multiple files are consistent, the method further includes:
and respectively entering a plurality of files, acquiring the md5 value of the locally saved file and copying the value to the/home directory of the Master host.
Further, the specific step of judging whether the distributed storage system is normal according to the judgment result is as follows:
if the md5 values of the files are consistent, the distributed storage system is normal, and if the md5 values are inconsistent, the distributed storage system is abnormal;
and outputting a test result.
The above summary provides merely a representation of embodiments of the invention and is not intended to be exhaustive or to limit the invention to the precise form disclosed.
The effect provided in the summary of the invention is only the effect of the embodiment, not all the effects of the invention, and one of the above technical solutions has the following advantages or beneficial effects:
the compatibility of the server and the distributed storage system platform can be verified, the problem of the compatibility can be found in advance, and the stability and the error rate of the server carrying the hadoop distributed storage system can be detected to detect the overall performance and the stability of the server
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate exemplary embodiments of the invention and, together with the description, serve to explain the invention and not to limit the invention. In the drawings:
FIG. 1 is a schematic flow chart of a method according to an embodiment of the present invention.
Detailed Description
In order to clearly explain the technical features of the present invention, the following detailed description of the present invention is provided with reference to the accompanying drawings. The following disclosure provides many different embodiments, or examples, for implementing different features of the invention. To simplify the disclosure of the present invention, the components and arrangements of specific examples are described below. Furthermore, the present invention may repeat reference numerals and/or letters in the various examples. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed. It should be noted that the components illustrated in the figures are not necessarily drawn to scale. Descriptions of well-known components and processing techniques and procedures are omitted so as to not unnecessarily limit the invention.
As shown in fig. 1, the method for Hadoop cluster testing performs read-write operation and file verification on files with different sizes of a distributed storage system, and detects whether a server product is matched with the Hadoop distributed storage system to function normally, specifically including the following steps:
acquiring md5 values of files with different capacities;
deleting the file which has acquired the md5 value;
acquiring the md5 value of the new file;
comparing the md5 value of the newly acquired file with the md5 value of the original file, and judging whether the file is normal or not according to the comparison result; if the result is normal, performing the next operation, otherwise, stopping the test and outputting the result;
saving the file as a plurality of copies;
and judging whether the md5 values of the files are consistent or not, and judging whether the distributed storage system is normal or not according to the judgment result.
All files under/home/hadoop beginning with file are required to be deleted before the md5 values of files of different sizes are obtained. Such as: rm-rf/home/hadoop/file.
Files for which the md5 value has been obtained include files under/home/hadoop beginning with md5sum, file- $ imd5is $ md5sum $ i.
While deleting the file with the md5 value acquired, the file beginning with the file under the/home needs to be deleted. Prevent the influence of the residual document on the test result. Such as:
rm-rf/home/file*
rm–rf/home/hadoop/md5sum*
the created file needs to be uploaded to hadoop and downloaded from hadoop to the local/home directory before the md5 value of the new file is obtained.
Comparing the md5 value of the newly acquired file with the md5 value of the original file, and judging whether the system function is normal according to the comparison result:
if the md5 value of the newly acquired file is the same as the md5 value of the original file, the system functions normally;
if the md5 value of the newly acquired file is not the same as the md5 value of the original file, it indicates that the system is not functioning properly.
Before judging whether the md5 values of the files are consistent, the method further comprises the following steps:
and respectively entering a plurality of files, acquiring the md5 value of the locally saved file and copying the value to the/home directory of the Master host.
Judging whether the distributed storage system is normal according to the judgment result specifically comprises the following steps:
if the md5 values of the files are consistent, the distributed storage system is normal, and if the md5 values are inconsistent, the distributed storage system is abnormal;
and outputting a test result.
The foregoing is only a preferred embodiment of the present invention, and it will be apparent to those skilled in the art that various modifications and improvements can be made without departing from the principle of the invention, and such modifications and improvements are also considered to be within the scope of the invention.
Claims (7)
- The Hadoop cluster testing method is characterized by performing read-write operation and file verification on files with different sizes of a distributed storage system, and detecting whether functions of a server product matched with the Hadoop distributed storage system are normal or not, and specifically comprises the following modes:acquiring md5 values of files with different capacities;deleting the file which has acquired the md5 value;uploading the created file to a hadoop, and downloading the file from the hadoop to a local/home directory;acquiring the md5 value of the new file;comparing the md5 value of the newly acquired file with the md5 value of the original file, and judging whether the file is normal or not according to the comparison result;if the result is normal, performing the next operation, otherwise, stopping the test and outputting the result;saving the file as a plurality of copies;and judging whether the md5 values of the files are consistent or not, and judging whether the distributed storage system is normal or not according to the judgment result.
- 2. The method of claim 1, wherein all files under/home/hadoop beginning with file are deleted before md5 values for files of different sizes are obtained.
- 3. The method of claim 1, wherein the files for which md5 values have been obtained include files beginning with md5sum under/home/hadoop.
- 4. A method as claimed in claim 3, wherein the file under/home beginning with file is deleted simultaneously with the deletion of the file for which md5 value has been obtained.
- 5. The method as claimed in claim 1, wherein the md5 value of the newly acquired file is compared with the md5 value of the original file, and the determination of whether the system function is normal according to the comparison result is specifically as follows:if the md5 value of the newly acquired file is the same as the md5 value of the original file, the system functions normally;if the md5 value of the newly acquired file is not the same as the md5 value of the original file, it indicates that the system is not functioning properly.
- 6. The method of claim 1, wherein determining whether the md5 values of the plurality of files match further comprises:and respectively entering a plurality of files, acquiring the md5 value of the locally saved file and copying the value to the/home directory of the Master host.
- 7. The method as claimed in claim 6, wherein the determining whether the distributed storage system is normal according to the determination result is specifically:if the md5 values of the files are consistent, the distributed storage system is normal, and if the md5 values are inconsistent, the distributed storage system is abnormal;and outputting a test result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711270115.2A CN108009071B (en) | 2017-12-05 | 2017-12-05 | Hadoop cluster test method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711270115.2A CN108009071B (en) | 2017-12-05 | 2017-12-05 | Hadoop cluster test method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108009071A CN108009071A (en) | 2018-05-08 |
| CN108009071B true CN108009071B (en) | 2020-10-20 |
Family
ID=62056912
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711270115.2A Active CN108009071B (en) | 2017-12-05 | 2017-12-05 | Hadoop cluster test method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108009071B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109976957B (en) * | 2019-02-11 | 2022-04-15 | 平安科技（深圳）有限公司 | Librgw performance test method and device and computer equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101272275A (en) * | 2008-05-07 | 2008-09-24 | 华为技术有限公司 | Network management equipment, method and system for acquiring network appliance configuration information |
| CN104298933A (en) * | 2014-10-17 | 2015-01-21 | 浪潮(北京)电子信息产业有限公司 | Configuration information safety processing method and system |
| CN105975865A (en) * | 2016-05-27 | 2016-09-28 | 北京工业大学 | Hadoop platform measuring method based on dependable computing |
| CN106059825A (en) * | 2016-07-08 | 2016-10-26 | 福建天晴数码有限公司 | Distributed system and configuration method |
| CN107315668A (en) * | 2017-06-26 | 2017-11-03 | 郑州云海信息技术有限公司 | Distributed memory system data consistency automates quick determination method and device |
-
2017
- 2017-12-05 CN CN201711270115.2A patent/CN108009071B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101272275A (en) * | 2008-05-07 | 2008-09-24 | 华为技术有限公司 | Network management equipment, method and system for acquiring network appliance configuration information |
| CN104298933A (en) * | 2014-10-17 | 2015-01-21 | 浪潮(北京)电子信息产业有限公司 | Configuration information safety processing method and system |
| CN105975865A (en) * | 2016-05-27 | 2016-09-28 | 北京工业大学 | Hadoop platform measuring method based on dependable computing |
| CN106059825A (en) * | 2016-07-08 | 2016-10-26 | 福建天晴数码有限公司 | Distributed system and configuration method |
| CN107315668A (en) * | 2017-06-26 | 2017-11-03 | 郑州云海信息技术有限公司 | Distributed memory system data consistency automates quick determination method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108009071A (en) | 2018-05-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10379831B2 (en) | Dynamic retrieval of installation packages when installing software | |
| US10437703B2 (en) | Correlation of source code with system dump information | |
| CN106775602B (en) | Code publishing method and device | |
| CN102736978B (en) | A kind of method and device detecting the installment state of application program | |
| CN108121559B (en) | Configuration file pushing method, server and storage medium | |
| CN110515795B (en) | Big data component monitoring method and device and electronic equipment | |
| CN109446753A (en) | Detect method, apparatus, computer equipment and the storage medium of pirate application program | |
| WO2015131643A1 (en) | Software detection method and device | |
| CN108009071B (en) | Hadoop cluster test method | |
| US10423495B1 (en) | Deduplication grouping | |
| US9720669B2 (en) | Deployment of software across an enterprise system | |
| WO2014059156A1 (en) | Efficient virtual machine deployment methodology | |
| EP3051457A1 (en) | Method for performing file synchronization control, and associated apparatus | |
| CN106657316B (en) | Message withdrawing method and device | |
| CN112148597B (en) | Method for eliminating iOS device authorization dialog box, test method and storage medium | |
| CN108959604B (en) | Method, apparatus and computer readable storage medium for maintaining database cluster | |
| CN104317645A (en) | Method and device of monitoring port by virtue of application program embodiment | |
| CN106557379B (en) | Method and device for generating silent installation package | |
| US20170322789A1 (en) | System and method for automatically installing application | |
| CN109960515B (en) | Image file making method and device | |
| CN112698842A (en) | Method and device for acquiring additional information of application program | |
| US11811548B2 (en) | System and method for appliance configuration identification and profile management | |
| CN112035119B (en) | Data deleting method and device | |
| US11341107B1 (en) | Verification microservice for execution of a slice verification path by worker nodes | |
| US20210064572A1 (en) | Information processing apparatus, information processing system, and non-transitory computer readable medium storing program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200917 Address after: 215100 No. 1 Guanpu Road, Guoxiang Street, Wuzhong Economic Development Zone, Suzhou City, Jiangsu Province Applicant after: SUZHOU LANGCHAO INTELLIGENT TECHNOLOGY Co.,Ltd. Address before: 450018 Henan province Zheng Dong New District of Zhengzhou City Xinyi Road No. 278 16 floor room 1601 Applicant before: ZHENGZHOU YUNHAI INFORMATION TECHNOLOGY Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |