CN209879421U

CN209879421U - Server equipment and server device based on HGX platform

Info

Publication number: CN209879421U
Application number: CN201920871103.3U
Authority: CN
Inventors: 王世锋; 李永伟; 符庆明
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Priority date: 2019-06-11
Filing date: 2019-06-11
Publication date: 2019-12-31
Anticipated expiration: 2029-06-11

Abstract

The utility model relates to a server equipment and server device based on HGX platform, this server equipment based on HGX platform include quick-witted case, HGX GPU module, P-Switch module and fan module, and the machine case has first end and second end that back to back, and HGX GPU module locates first end, and P-Switch module locates the second end, P-Switch module with quick-witted case and HGX GPU module all can be dismantled and be connected, the fan module is located on the Switch board of P-Switch module. According to the HGX platform-based server equipment, the P-Switch module and the case as well as the HGX GPU module can be detachably connected, and the maintenance is convenient.

Description

Server equipment and server device based on HGX platform

Technical Field

The present disclosure relates to the field of server technologies, and in particular, to a server device and a server apparatus based on an HGX platform.

Background

With the increasing popularization of networks and the rapid development of new technologies such as cloud computing and big data, a large number of server devices are required for data backup of individuals and data platforms of enterprises. At present, the existing servers supporting the GPU all realize the increase of the number of GPU cards by adding GPU AIC cards to the existing system. However, this approach has a limited number of GPU cards to support and is more costly.

HGX-2 can provide unparalleled computational performance, and HGX-2 is provided with NVIDIA NVSwitch^TMAnd 16 GPUs are seamlessly connected together to form a huge GPU by using breakthrough functions such as an interconnection structure and the like, and the huge GPU can provide 2 trillion AI performance. The HGX-2 cloud server platform has multi-precision computing capability, can provide incomparable flexibility to support future computing, and can realize accurate high-performance computing, deep learning and machine learning.

However, the existing HGX-2 server has high maintenance difficulty and high cost.

It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure, and thus may include information that does not constitute prior art known to those of ordinary skill in the art.

Disclosure of Invention

The invention aims to provide server equipment and a server device based on an HGX platform, which are convenient to maintain.

According to an aspect of the present disclosure, there is provided an HGX platform-based server apparatus including:

the computer case is provided with a first end and a second end which are opposite;

the HGX GPU module is arranged at the first end;

the P-Switch module is arranged at the second end, and the P-Switch module can be detachably connected with the case and the HGX GPU module;

and the fan module is arranged on the Switch plate of the P-Switch module.

In an exemplary embodiment of the present disclosure, the server apparatus further includes:

the water-cooling module comprises a water-cooling radiator, a second water inlet pipe and a second water outlet pipe which are arranged on the HGX GPU module, and a first water inlet pipe and a first water outlet pipe which are arranged on the P-Switch module;

wherein, the second inlet tube with the one end of second outlet pipe with water-cooled radiator connects, the other end of second inlet tube with the connection can be dismantled to first inlet tube, the other end of second outlet pipe with the connection can be dismantled to first outlet pipe.

In an exemplary embodiment of the present disclosure, a plurality of heat sinks are provided, and are disposed on the GPUs of the HGX GPU modules in a one-to-one correspondence; wherein,

a plurality of the radiators are connected in series through a connecting pipe; or,

the radiators are connected with the second water outlet pipes in parallel through the second water inlet pipes; or,

the radiators are connected in series-parallel with the second water outlet pipes through a plurality of connecting pipes and a plurality of second water inlet pipes.

In an exemplary embodiment of the present disclosure, the water cooling module further includes:

the water distributor is provided with interfaces corresponding to the quantity of the second water outlet pipes and the second water inlet pipes and is used for being connected with the second water outlet pipes and the second water inlet pipes; the water distributor is also provided with a first water outlet connector and a first water inlet connector which face the second end, the first water inlet connector is detachably connected with the first water inlet pipe, and the first water outlet connector is detachably connected with the first water outlet pipe.

the adapter is arranged on the P-Switch module and comprises a second water inlet connector, a second water outlet connector, a third water inlet connector and a third water outlet connector, the second water inlet connector is used for being connected with the first water inlet connector in an inserting mode, the second water outlet connector is connected with the first water outlet connector in an inserting mode, the third water inlet connector is connected with the first water inlet pipe, and the third water outlet connector is connected with the first water outlet pipe.

In an exemplary embodiment of the present disclosure, the fan module includes a plurality of fans and a bracket, the bracket is formed with a plurality of mounting portions, and each of the fans can be disposed in each of the mounting portions in a one-to-one correspondence; the adapter is matched with the mounting part in shape and size and can be mounted in the mounting part.

In an exemplary embodiment of the disclosure, a GPU board connector is disposed on a side of the P-Switch module close to the first end, and the P-Switch module is detachably connected to the HGX GPU module through the GPU board connector.

In an exemplary embodiment of the disclosure, a Switch board connector is disposed on a side of the HGX GPU module near the second end, and the Switch board connector is used for plugging with the GPU board connector.

According to another aspect of the present disclosure, there is also provided a server apparatus including:

two server devices based on HGX platforms as described in any of the above embodiments;

the casing has the open cavity in one both ends, be equipped with a baffle in the cavity, the baffle will the cavity is split into the open sub cavity in two both ends, two server equipment one-to-one based on HGX platform is located in the sub cavity.

the panel is arranged on one open end of the shell and corresponds to the first end position of the HGX platform-based server equipment, and a plurality of positioning holes are formed in the panel;

and each HGX platform-based server device is provided with positioning grooves corresponding to the positioning holes in number and position on the first end, and the positioning pins are used for being inserted into the positioning grooves through the positioning holes.

The server equipment based on the HGX platform provided by the disclosure is compatible with 4U and 8U models in the novel GPU server design based on the HGX GPU board, can meet the deployment of 8 GPUs, and the P-Switch module, the case and the HGX GPU module can be detachably connected. The fan module is arranged on the Switch board, when the fan module needs to be maintained, the P-Switch module can be pulled out from the second end, the fan module is replaced or maintained, and maintenance of all parts in the server equipment is facilitated.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure. It is to be understood that the drawings in the following description are merely exemplary of the disclosure, and that other drawings may be derived from those drawings by one of ordinary skill in the art without the exercise of inventive faculty.

FIG. 1 is a schematic diagram of a server device based on an HGX platform provided by an embodiment of the present disclosure;

FIG. 2 is a front view of FIG. 1;

FIG. 3 is a schematic diagram of the enclosure of FIG. 1 with the upper cover opened;

FIG. 4 is an exploded view of FIG. 3;

FIG. 5 is a schematic diagram of a server device based on an HGX platform according to another embodiment of the present disclosure;

FIG. 6 is a top view of FIG. 5;

FIG. 7 is a schematic diagram of the P-Switch module of FIG. 6 shown in a pulled-out state;

FIG. 8 is a schematic diagram of the HGX GPU module of FIG. 7;

FIG. 9 is a diagram of the P-Switch module of FIG. 7;

FIG. 10 is a schematic view of an adapter provided in an embodiment of the present disclosure;

fig. 11 is a schematic diagram of a server apparatus provided by an embodiment of the present disclosure;

FIG. 12 is a front view of FIG. 11;

fig. 13 is an exploded view of fig. 11:

fig. 14 is a schematic view of a locating pin provided in one embodiment of the present disclosure.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar structures, and thus their detailed description will be omitted.

Although relative terms, such as "upper" and "lower," may be used in this specification to describe one element of an icon relative to another, these terms are used in this specification for convenience only, e.g., in accordance with the orientation of the examples described in the figures. It will be appreciated that if the device of the icon were turned upside down, the element described as "upper" would become the element "lower". When a structure is "on" another structure, it may mean that the structure is integrally formed with the other structure, or that the structure is "directly" disposed on the other structure, or that the structure is "indirectly" disposed on the other structure via another structure.

The terms "a," "an," "the," "said," and "at least one" are used to indicate the presence of one or more elements/components/parts/etc.; the terms "comprising" and "having" are intended to be inclusive and mean that there may be additional elements/components/etc. other than the listed elements/components/etc.; the terms "first," "second," and "third," etc. are used merely as labels, and are not limiting on the number of their objects.

The embodiment of the present disclosure provides a server device based on an HGX platform, as shown in fig. 1 to 10, the server device based on the HGX platform includes a chassis 11, an HGX GPU module 40, a P-Switch module 30, and a fan module 50, the chassis 11 has a first end and a second end opposite to each other, the HGX GPU module 40 is disposed at the first end, the P-Switch module 30 is disposed at the second end, the P-Switch module 30 is detachably connected to both the chassis 11 and the HGX GPU module 40, and the fan module 50 is disposed on a Switch board 301 of the P-Switch module 30.

The novel HGX platform-based server equipment provided by the disclosure is compatible with 4U and 8U models based on the design of a HGX GPU server board 401, can meet the deployment of 8 GPUs, and is detachably connected with a P-Switch module 30, a case 11 and an HGX GPU module 40. The fan module 50 is arranged on the Switch board 301, when the fan module 50 needs to be maintained, the P-Switch module 30 can be pulled out from the second end, and the fan module 50 can be replaced or maintained, so that maintenance of each component in the server equipment is facilitated.

Specifically, the fan module 50 is detachably disposed on the P-Switch module 30 near one end of the HGX GPU module 40. Through the arrangement of the fan module 50, heat generated during the operation of the HGX GPU module 40 can be timely exhausted out of the server equipment, and the overhigh temperature in the case 11 of the server equipment is avoided. By arranging the fan module 50 on the P-Switch module 30 near one end of the HGX GPU module 40, the fan module 50 is closer to the heat source, and the heat removal efficiency of the fan module 50 is improved.

As shown in fig. 5, the server device further includes a water cooling module 60, where the water cooling module 60 includes a heat exchange pipeline and a water cooling heat sink 601 disposed on the HGX GPU module 40, one end of the heat exchange pipeline is connected to the water cooling heat sink 601, and the other end of the heat exchange pipeline extends out of the second end for being connected to a heat exchange device outside the server. Through the arrangement of the water-cooling module 60, the heat generated by each GPU chip on the HGX GPU module 40 can be absorbed in time through the water-cooling radiator 601, and then the heat absorbed by the water-cooling radiator 601 is transmitted to the heat exchange equipment outside the server equipment through the heat exchange pipeline for heat dissipation.

Because the GPU has a high power, only the fan module 50 is used for heat dissipation, and the energy consumption in the heat dissipation process is high, so that the TCO (Total Cost of Ownership) is high, and the air cooling module has a certain noise, and by the arrangement of the water cooling module 60, the energy consumption generated by heat dissipation can be reduced, the operating Cost and the maintenance Cost of the server equipment can be reduced, and no noise is generated.

In an embodiment, the heat exchange device may be an air-cooled radiator connected to the heat exchange pipeline, a flow channel communicated with the heat exchange pipeline is provided in the air-cooled radiator, the heat exchange liquid can flow back to the air-cooled radiator, the heat exchange pipeline and the water-cooled radiator in a circulating manner, heat dissipation fins are provided on an outer surface of the air-cooled radiator to increase a heat exchange area of the radiator, and then an air flow rate of the outer surface of the air-cooled radiator is increased by using a fan to increase a cooling efficiency of the liquid change in the air-cooled radiator. Those skilled in the art can also use other heat exchange devices to reduce the temperature of the heat exchange liquid in the heat exchange pipeline, for example, a compressor is used to reduce the temperature of the heat exchange liquid, which is not limited in this disclosure, and all the changes of the heat exchange device belong to the protection scope of this application.

The heat exchange liquid can be water, the water is used as the liquid exchange liquid, the heat exchange capacity is strong, the cost is low, the maintenance of the heat exchange assembly is convenient, and the liquid exchange liquid is not limited by the liquid exchange liquid.

Specifically, the heat exchange pipeline includes a first water inlet pipe 607, a first water outlet pipe 608, a second water inlet pipe 604 and a second water outlet pipe 603, the first water inlet pipe 607 and the first water outlet pipe 608 are arranged on the P-Switch module 30, the second water inlet pipe 604 and the second water outlet pipe 603 are arranged on the HGX GPU module 40, one end of the second water inlet pipe 604 and one end of the second water outlet pipe 603 are connected with the heat exchanger, the other end of the second water inlet pipe 604 is detachably connected with the first water inlet pipe 607, and the other end of the second water outlet pipe 603 is detachably connected with the first water outlet pipe 608. Through the arrangement of the first water inlet pipe 607, the first water outlet pipe 608, the second water inlet pipe 604 and the second water outlet pipe 603, when the P-Switch module 30 needs to be pulled out from the chassis 11, heat exchange liquid in a heat exchange pipeline can be discharged from the first water inlet pipe 607, the first water inlet pipe 607 and the first water outlet pipe 608 are fixedly arranged on the P-Switch module 30, along with the pulling out of the P-Switch module 30, the first water outlet pipe 608 is disconnected from the second water outlet pipe 603, and the first water inlet pipe 607 is disconnected from the second water inlet pipe 604, so that the P-Switch module 30 can be detachably mounted.

Further, the water-cooling module 60 is provided with a plurality of heat sinks 601, which are correspondingly disposed on the GPUs of the HGX GPU module 40. Wherein, a plurality of radiators 601 are connected in series by a connection pipe 602; or, the radiators 601 are connected in parallel with the second water outlet pipes 603 through the second water inlet pipes 604; alternatively, the radiators 601 are connected in series and parallel with the second outlet pipes 603 through the connecting pipes 602 and the second inlet pipes 604.

When a plurality of radiators 601 are connected in series through a connecting pipe 602, liquid for replacing enters a water-cooling radiator 601 through a water inlet pipe for connection, then the plurality of water-cooling radiators 601 are connected in series one by one through the connecting pipe 602, and then a water outlet pipe is arranged on the last water-cooling radiator 601; when the radiators 601 are connected in parallel with the second water outlet pipes 603 through the second water inlet pipes 604, each water-cooled heat exchanger is provided with at least one water inlet pipe and one water outlet pipe; when the radiators 601 are connected in series and parallel with the second water outlet pipes 603 through the connecting pipes 602 and the second water inlet pipes 604, part of the water-cooled radiators 601 can be connected in series through the connecting pipes 602, then the circulation of the liquid change liquid is realized through the water inlet pipes and the water outlet pipes, the plurality of water-cooled radiators 601 connected in series through the connecting pipes 602 form a radiator 601 group, and the plurality of radiator 601 groups are connected in parallel through the water inlet pipes and the water outlet pipes, so that the mixed connection of the plurality of water-cooled fuel heaters is realized.

As shown in fig. 8, eight water-cooled radiators 601 are provided, two water-cooled radiators 601 are connected in series to form four radiator 601 groups, and then the parallel connection of the four radiator 601 groups is realized through four second water inlet pipes 604 and four second water outlet pipes 603. The plurality of water-cooled radiators 601 are connected in a mixed mode, so that the arrangement of heat exchange pipelines is facilitated, the lengths between the water outlet pipes and the lengths between the water inlet pipes are ensured to be as consistent as possible, and the heat dissipation effect on each GPU is ensured.

As shown in fig. 8, the water cooling module 60 further includes a water distributor 605, the water distributor 605 is provided with interfaces corresponding to the number of the second water outlet pipe 603 and the second water inlet pipe 604, and is used for connecting the second water outlet pipe 603 and the second water inlet pipe 604; the water separator 605 is further provided with a first water outlet joint 6052 and a first water inlet joint 6051 facing the second end, the first water inlet joint 6051 is detachably connected with the first water inlet pipe 607, and the first water outlet joint 6052 is detachably connected with the first water outlet pipe 608. Through the arrangement of the water separator 605, a plurality of second water inlet pipes 604 and second water outlet pipes 603 can be collected together, so as to be conveniently connected with the first water inlet pipe 607 and the first water outlet pipe 608.

As shown in fig. 9 and 10, the water-cooling module 60 further includes an adapter 606, the adapter 606 is disposed on the P-Switch module 30, the adapter 606 includes a second water inlet joint 6062, a second water outlet joint 6061, a third water inlet joint 6063 and a third water outlet joint 6064, the second water inlet joint 6061 is used for being plugged with the first water inlet joint 6051, the second water outlet joint 6062 is plugged with the first water outlet joint 6052, the third water inlet joint 6063 is connected with the first water inlet pipe 607, and the third water outlet joint 6064 is connected with the first water outlet pipe 608. Through the arrangement of the adapter 606 and the water distributor 605, the detachable connection between the first water outlet pipe 603 and the second water outlet pipe 608, and between the first water inlet pipe 603 and the second water inlet pipe 607 can be realized, the sealing connectivity still can be ensured after multiple splicing and separation, and the leakage of heat exchange liquid from the connection part is avoided, so that the reliability of the detachable installation of the P-Switch module 30 on the case 11 is ensured.

Specifically, the fan module 50 includes a plurality of fans and a bracket, the bracket is formed with a plurality of mounting portions, and the fans can be correspondingly arranged in the mounting portions one by one; adapter 606 matches with the shape and the size of installation department, can install on the installation department, when needs carry out the water-cooling, gets rid of a fan, then with adapter 606 install get rid of the fan on the position can, simple structure, simple to operate need not reset the structure of installation adapter 606.

In one embodiment, as shown in fig. 6, the fan module 50 has five sets of fans, and the fan module 50 is provided with a P-Switch module 30. When the adapter 606 is installed, the third group of fans in the fan module 50, i.e. the middle fan, is removed, and then the adapter 606 is installed at this position, so that the purpose that the fan module 50 and the water cooling module 60 can perform heat dissipation simultaneously is achieved. Of course, the adaptor 606 may be installed at other positions, for example, the first set of fans, or a mounting portion for mounting the adaptor 606 is reserved on the fan module 50, so as to mount the adaptor 606. Wherein, a set of fan's size and shape are the same in adapter 606 and the fan module 50, and when removing the third group fan, adapter 606 can be placed on the position of third group fan, can with adapter 606 direct mount in fan module 50's fan bracket, the installation of adapter 606 of being convenient for need not to set up other structures and carries out the installation of adapter 606 for server equipment's structure can be compacter.

As shown in fig. 9, a water inlet quick connector is disposed at one end of the first water inlet pipe 607 extending out of the second end, a water outlet quick connector is disposed at one end of the first water outlet pipe 608 extending out of the second end, and the arrangement of the water inlet quick connector and the water outlet quick connector facilitates connection between the heat exchange pipeline and external heat exchange equipment, and also facilitates detachment and installation of the P-Switch module 30.

Specifically, a GPU board 401 connector is disposed on a side of the P-Switch module 30 close to the first end, and the P-Switch module 30 is detachably connected to the HGX GPU module 40 through the GPU board 401 connector.

Further, a Switch board 301 connector is arranged on one side of the HGX GPU module 40 close to the second end, and the Switch board 301 connector is used for plugging with a GPU board 401 connector.

Through the arrangement of the GPU board 401 connector and the Switch board 301 connector, the P-Switch board 301 and the GPU board 401 can be conveniently plugged, the connection and the separation of the P-Switch module 30 and the HGX GPU module 40 can be conveniently realized, and meanwhile, the P-Switch module 30 can still have good connectivity with the HGX GPU module 40 after being disassembled and installed for many times, so that the reliability of the detachable installation of the P-Switch module 30 on the case 11 can be further guaranteed.

The present disclosure further provides a server apparatus, as shown in fig. 11 to 13, the server apparatus includes a housing 21 and two above-mentioned server devices based on the HGX platform, the housing 21 has a cavity with two open ends, a partition board is disposed in the cavity, the cavity is divided into two sub-cavities with two open ends by the partition board, and the two server devices based on the HGX platform are disposed in the sub-cavities in a one-to-one correspondence manner.

Wherein, the size and the shape of sub-cavity and the size and the shape of the server equipment based on HGX platform match, avoid the server equipment based on HGX platform to rock or the condition of strutting the casing appears in the sub-cavity.

Specifically, usually, the two sides corresponding to the first end of the server equipment based on the HGX platform are provided with the hangers, and before the server equipment based on the HGX platform is arranged in the sub-cavity, the two sides corresponding to the first end of the server equipment based on the HGX platform are provided with the hangers to be removed, so that the influence on the installation of the server equipment of the HGX platform on the shell is avoided. In addition, the server device is provided with left and right hangers 25 correspondingly arranged on the outer side of the shell, and handles, control keys, indicator lights and the like are arranged on the hangers 25.

The server device provided by the disclosure can meet the design of GPU servers with a preset number by forming the sub-cavities through the shell, for example, when the cavity is divided into two sub-cavities with the same size and two server devices based on HGX platforms are accommodated, the requirement that the server device has 16 GPUs can be met to the maximum extent.

As shown in fig. 12 to 14, the server apparatus further includes a first panel 22 and positioning pins 23, the first panel 22 is disposed on an open end of the casing 21 and corresponds to a first end of the server device based on the HGX platform, and the first panel 22 is provided with a plurality of positioning holes; each server device based on the HGX platform is provided with a second panel 12 on a first end, the second panel 12 is provided with positioning slots 13 corresponding to the number and positions of the positioning holes, and the positioning pins 23 are used for being inserted into the positioning slots 13 through the positioning holes.

Specifically, be equipped with NVLink module 24 on the first panel 22, through the setting of locating pin 23, locating hole and constant head tank 13, can guarantee on the first panel 22 that the connector of NVLink module 24 is adjusted well with the connector position on the GPU board 401, the equipment of the server device of being convenient for. As shown in fig. 12, eight positioning pins are arranged and distributed on the first panel 22 in two rows and four columns; the locating pin 23 is T-shaped, the end part of the locating pin is inserted into the locating groove 13 and clamped with the locating groove 13, and when the locating pin needs to be detached, the locating pin can be pulled out, and the installation and the detachment are convenient. The number, distribution and shape of the positioning pins are not limited in the disclosure, and all schemes capable of achieving the same technical effect belong to the protection scope of the disclosure.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

Claims

1. A server device based on an HGX platform, comprising:

the HGX GPU module is arranged at the first end;

and the fan module is arranged on the Switch plate of the P-Switch module.

2. The server device according to claim 1, characterized in that the server device further comprises:

3. The server apparatus according to claim 2, wherein a plurality of the heat sinks are provided, one for one on the GPUs of the HGX GPU modules; wherein,

4. The server apparatus of claim 3, wherein the water-cooling module further comprises:

5. The server apparatus of claim 4, wherein the water-cooling module further comprises:

6. The server apparatus according to claim 5, wherein the fan module includes a plurality of fans and a bracket, the bracket is formed with a plurality of mounting portions, and each of the fans is provided in each of the mounting portions in a one-to-one correspondence; the adapter is matched with the mounting part in shape and size and can be mounted in the mounting part.

7. The server apparatus according to claim 1, wherein a GPU board connector is disposed on a side of the P-Switch module near the first end, and the P-Switch module is detachably connected to the HGX GPU module through the GPU board connector.

8. The server apparatus according to claim 7, wherein a Switch board connector is disposed on a side of the HGX GPU module near the second end, and the Switch board connector is configured to be plugged with the GPU board connector.

9. A server apparatus, comprising:

two HGX platform based server devices according to any of claims 1 to 8;

10. The server apparatus according to claim 9, characterized in that the server apparatus further comprises:

the first panel is arranged on one open end of the shell and corresponds to the first end position of the HGX platform-based server equipment, and a plurality of positioning holes are formed in the panel;

and each HGX platform-based server device is provided with a second panel on the first end, the second panel is provided with positioning grooves corresponding to the positioning holes in number and position, and the positioning pins are used for being inserted into the positioning grooves through the positioning holes.