US20180239635A1 - Suspend and restore processor operations - Google Patents
Suspend and restore processor operations Download PDFInfo
- Publication number
- US20180239635A1 US20180239635A1 US15/438,466 US201715438466A US2018239635A1 US 20180239635 A1 US20180239635 A1 US 20180239635A1 US 201715438466 A US201715438466 A US 201715438466A US 2018239635 A1 US2018239635 A1 US 2018239635A1
- Authority
- US
- United States
- Prior art keywords
- work
- items
- queues
- units
- flight
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/485—Task life-cycle, e.g. stopping, restarting, resuming execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/461—Saving or restoring of program or task context
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
Definitions
- a graphics processing unit is a complex integrated circuit that is configured to perform graphics-processing tasks.
- a GPU can execute graphics-processing tasks required by an end-user application, such as a video-game application. GPUs are also increasingly being used to perform other tasks which are unrelated to graphics.
- the GPU can be a discrete device or can be included in the same device as another processor, such as a central processing unit (CPU).
- a sequence of work-items which can also be referred to as threads, are processed so as to output a final result.
- processors within a single instruction multiple data (SIMD) core synchronously execute a set of work-items.
- SIMD single instruction multiple data
- the synchronous executing of work-items are identical (i.e., have the identical code base).
- a plurality of identical synchronous work-items that are processed by separate processors are referred to as a wavefront or warp.
- one or more SIMD cores concurrently execute multiple wavefronts. Execution of the wavefront terminates when all work-items within the wavefront complete processing.
- Each wavefront includes multiple work-items that are processed in parallel, using the same set of instructions.
- the time required for each work-item to complete processing depends on a criterion determined by data within the work-item. As such, the work-items with the wavefront can complete processing at different times.
- a user requests to suspend operation of the physical machine which includes the GPU, there are typically many in-flight operations executing on the GPU. Determining how to suspend operations on the GPU to enable a seamless restoration can be challenging. Accordingly, improved techniques for implementing suspend and restore operations on a processor are desired.
- FIG. 1 is a block diagram of one embodiment of a computing system.
- FIG. 2 is a block diagram of one embodiment of a GPU.
- FIG. 3 is a block diagram of one embodiment of a parallel processing environment.
- FIG. 4 illustrates one embodiment of a stored state.
- FIG. 5 is a generalized flow diagram illustrating one embodiment of a method for performing a suspend operation of in-flight work-items on a processor.
- FIG. 6 is a generalized flow diagram illustrating one embodiment of a method for performing a restore operation for a previously suspended processor state.
- a processor includes at least a plurality of execution units, a plurality of work creation units, and a control unit coupled to the plurality of execution units and the plurality of work creation units.
- the control unit in response to detecting a request to suspend a software application executing on the processor, the control unit is configured to send requests to the plurality of work creation units to stop creating new work. The control unit waits until receiving acknowledgements in response to the requests from the plurality of work creation units prior to initiating a suspend operation. Once all work creation units have acknowledged that they will stop creating new work, the control unit initiates the suspend operation.
- initiating the suspend operation involves determining which work-items are in-flight, determining which work creation units initiated the work-items, determining which execution units have been allocated for the in-flight work-items, and storing status information about the in-flight work items and corresponding execution units.
- the status information includes which work-items are in-flight, identifiers (IDs) of work creation units that initiated the work-items, IDs of the execution units that have been allocated for the in-flight work-items, and additional information.
- initiating the suspend operation includes one or more other steps.
- the work creation units are threads of the application and queues, and the execution units are single instruction, multiple data (SIMD) units.
- the control unit receives a request to restore a previously suspended state of the processor.
- the control unit empties the work creation units and causes the work creation units to be idle prior to starting a restore operation.
- the control unit also retrieves stored information, corresponding to the previously suspended state, about in-flight work items and corresponding work creation units and execution units.
- the control unit prevents the plurality of work creation units from launching new work-items. Then, the control unit restores in-flight work items to each work creation unit and corresponding execution units.
- each work-item is restored to a same work creation unit and/or a same execution unit on which the work-item was executing when the suspend operation occurred.
- computing system 100 includes system on chip (SoC) 105 coupled to memory 150 .
- SoC 105 can also be referred to as an integrated circuit (IC).
- SoC 105 includes processing units 115 A-N, input/output (I/O) interfaces 110 , shared caches 120 A-B, fabric 125 , graphics processing unit (GPU) 130 , and memory controller(s) 140 .
- SoC 105 can also include other components not shown in FIG. 1 to avoid obscuring the figure.
- Processing units 115 A-N are representative of any number and type of processing units.
- processing units 115 A-N are central processing unit (CPU) cores.
- processing units 115 A-N are other types of processing units (e.g., application specific integrated circuit (ASIC), field programmable gate array (FPGA), digital signal processor (DSP)).
- ASIC application specific integrated circuit
- FPGA field programmable gate array
- DSP digital signal processor
- Processing units 115 A-N are coupled to shared caches 120 A-B and fabric 125 .
- processing units 115 A-N are configured to execute instructions of a particular instruction set architecture (ISA). Each processing unit 115 A-N includes one or more execution units, cache memories, schedulers, branch prediction circuits, and so forth. In one embodiment, the processing units 115 A-N are configured to execute the main control software of system 100 , such as an operating system. Generally, software executed by processing units 115 A-N during use can control the other components of system 100 to realize the desired functionality of system 100 . Processing units 115 A-N can also execute other software, such as application programs.
- ISA instruction set architecture
- GPU 130 includes at least control unit 132 , queues 135 , and compute units 145 A-N which are representative of any number and type of compute units that are used for graphics or general-purpose processing. GPU 130 is coupled to shared caches 120 A-B and fabric 125 . In one embodiment, GPU 130 is configured to execute graphics pipeline operations such as draw commands, pixel operations, geometric computations, and other operations for rendering an image to a display. In another embodiment, GPU 130 is configured to execute operations unrelated to graphics. In a further embodiment, GPU 130 is configured to execute both graphics operations and non-graphics related operations.
- control unit 132 of GPU 130 is configured to initiate a suspend operation in response to receiving a request to suspend an application executing on GPU 130 .
- control unit 132 causes the work creation units (e.g., queues 135 ) to stop launching new work-items prior to initiating the suspend operation.
- control unit 132 starts a timer prior to initiating the suspend operation and sets a grace period for work creation units to become idle based on a number of clock cycles of the timer.
- control unit 132 waits to receive acknowledgements from all work creation units that these units have stopped creating new work.
- control unit 132 initiates the suspend operation to determine and record information about the in-flight work-items, the work creation units that launched the in-flight work-items, the execution units which have been allocated for the in-flight work-items, and additional status information.
- control unit 132 stores the information as stored state 155 in memory 150 . In other embodiments, control unit 132 can allocate stored state 155 in another location.
- control unit 132 of GPU 130 is configured to initiate a restore operation in response to receiving a request to restore a previously suspended application.
- control unit 132 retrieves status information (e.g., stored state 155 ) corresponding to the previously suspended application.
- status information e.g., stored state 155
- control unit 132 empties and idles all queues 135 and other work creation units.
- control unit 132 restores the in-flight work-items specified in the status information to the same queues 135 and execution units to which the work-items were previously allocated. After restoring all of the previously in-flight work-items, control unit 132 allows the work creation units to start launching new work-items.
- a “restore operation” can also be referred to as a “resume operation”.
- I/O interfaces 110 are coupled to fabric 125 , and I/O interfaces 110 are representative of any number and type of interfaces (e.g., peripheral component interconnect (PCI) bus, PCI-Extended (PCI-X), PCIE (PCI Express) bus, gigabit Ethernet (GBE) bus, universal serial bus (USB)).
- PCI peripheral component interconnect
- PCI-X PCI-Extended
- PCIE PCI Express
- GEE gigabit Ethernet
- USB universal serial bus
- peripheral devices include (but are not limited to) displays, keyboards, mice, printers, scanners, joysticks or other types of game controllers, media recording devices, external storage devices, network interface cards, and so forth.
- SoC 105 is coupled to memory 150 , which includes one or more memory modules. Each of the memory modules includes one or more memory devices mounted thereon. In some embodiments, memory 150 includes one or more memory devices mounted on a motherboard or other carrier upon which SoC 105 is also mounted. In one embodiment, memory 150 is used to implement a random access memory (RAM) for use with SoC 105 during operation.
- RAM random access memory
- the RAM implemented can be static RAM (SRAM), dynamic RAM (DRAM), Resistive RAM (ReRAM), Phase Change RAM (PCRAM), or any other volatile or non-volatile RAM.
- the type of DRAM that is used to implement memory 150 includes (but is not limited to) double data rate (DDR) DRAM, DDR2 DRAM, DDR3 DRAM, and so forth.
- SoC 105 can also include one or more cache memories that are internal to the processing units 115 A-N and/or compute units 145 A-N.
- SoC 105 includes shared caches 120 A-B that are utilized by processing units 115 A-N and compute units 145 A-N.
- caches 120 A-B are part of a cache subsystem including a cache controller.
- computing system 100 can be a computer, laptop, mobile device, server or any of various other types of computing systems or devices. It is noted that the number of components of computing system 100 and/or SoC 105 can vary from embodiment to embodiment. There can be more or fewer of each component/subcomponent than the number shown in FIG. 1 . For example, in another embodiment, SoC 105 can include multiple memory controllers coupled to multiple memories. It is also noted that computing system 100 and/or SoC 105 can include other components not shown in FIG. 1 . Additionally, in other embodiments, computing system 100 and SoC 105 can be structured in other ways than shown in FIG. 1 .
- GPU 200 includes command processor 205 , asynchronous compute engines (ACEs) 210 A-N, compute units 240 , cache 245 , memory controller 250 , and direct memory access (DMA) engines 255 and 260 . It is noted that GPU 200 also includes other logic and/or other components which are not shown in FIG. 2 to avoid obscuring the figure. It is also noted that the structure of GPU 200 is merely one example of a structure which is used in one embodiment. In other embodiments, GPU 200 can be organized in other manners.
- ACEs asynchronous compute engines
- DMA direct memory access
- the command processor 205 also includes a save/restore machine 212 .
- the save/restore machine 212 is (or effectively implements) a finite state machine which is triggered and managed by firmware and/or microcode.
- status information associated with the in-flight work-items is saved to memory by save/restore machine 212 .
- Data associated with the in-flight work items is streamed from different units of GPU 200 to save/restore machine 212 , and restore information identifying where the data came from is embedded alongside the data. For example, each piece of data sent to save/restore machine 212 is tagged to indicate which unit generated the piece of data.
- data is streamed over a common bus to save/restore machine 212 , and save/restore machine 212 uses the embedded tags to determine which unit(s) generated each piece of data.
- Save/restore machine 212 saves the status information and embedded tags to memory, and this stored data will be used later if a restore operation is requested.
- This restore information will be used by the save/restore machine 212 to redirect the saved data in memory back to the original source in response to a restore operation being launched.
- GPU 200 includes a plurality of queues for storing multiple work-items.
- command processor 205 includes queue 215
- ACE 210 A includes queues 220
- ACE 210 B includes queues 225
- ACE 210 N includes queues 230 .
- Queues 220 , 225 , and 230 are representative of any number and type of queues, with each queue including storage elements for storing a plurality of entries.
- the different types of queues can include graphics queues for primary rendering tasks, compute queues for supporting GPU tasks, copy queues for data transfers, and/or other types of queues.
- ACEs 210 A-N are representative of any number of asynchronous compute engines which are included in GPU 200 .
- ACEs 210 A-N are configured to parse incoming commands and dispatch work to the processing elements in compute units 240 .
- the processing elements in compute units 240 can also be referred to as “execution units”.
- command processor 205 handles graphics tasks
- ACEs 210 A-N handle compute tasks
- DMA engines 255 and 260 handle data transfer tasks.
- Each queue in GPU 200 is able to dispatch work items without waiting for other tasks to complete, allowing independent command streams to be interleaved in compute units 240 and execute simultaneously.
- parallel processing environment 300 includes a GPU.
- parallel processing environment 300 includes other types of processors (e.g., digital signal processor (DSP), field programmable gate array (FPGA), application specific integrated circuit (ASIC), multi-core processor).
- DSP digital signal processor
- FPGA field programmable gate array
- ASIC application specific integrated circuit
- Application 305 is representative of any number and type of software applications that are executing in parallel processing environment 300 .
- application 305 is a multi-threaded application, with application 305 including threads 315 A-N, which are representative of any number of threads.
- Suspend/restore control unit 345 includes logic for suspending and restoring the state of the processor.
- suspend/restore control unit 345 includes a save/restore machine (e.g., save/restore machine 212 of FIG. 2 ).
- suspend/restore control unit 345 can suspend and restore the state of the entire processor or suspend/restore control unit 345 can suspend and restore the state of a portion of the processor.
- Suspend/restore control unit 345 is implemented using any suitable combination of software and/or hardware, depending on the embodiment.
- suspend/restore control unit 345 will receive streamed data from various parts of the processor that manage work-items, and restore information will be embedded with the streamed data indicating where the work-items came from in the processor. Suspend/restore control unit 345 will save this data to memory, and this saved data will be used later if a restore operation is requested. The saved data will be used by the suspend/restore control unit 345 to redirect the saved data from memory back to the original source in response to a restore operation being launched.
- suspend/restore control unit 345 When suspend/restore control unit 345 detects a suspend request, suspend/restore control unit 345 prevents new work from being generated by the work creation units (e.g., threads 315 A-N, queues 320 A-N). In one embodiment, suspend/restore control unit 345 generates a request for application 305 to stop sending new work to queues 320 A-N. In one embodiment, suspend/restore control unit 345 also generates requests for queues 320 A-N to stop sending new work-items to execution units 350 A-N.
- the work creation units e.g., threads 315 A-N, queues 320 A-N
- suspend/restore control unit 345 When suspend/restore control unit 345 detects a suspend request, suspend/restore control unit 345 prevents new work from being generated by the work creation units (e.g., threads 315 A-N, queues 320 A-N). In one embodiment, suspend/restore control unit 345 generates a request
- suspend/restore control unit 345 starts to store the status of queues 320 A-N and execution units 350 A-N.
- the status includes information about work-items 325 A-D assigned to queue 320 A, work-items 330 A-B assigned to queue 320 B, work-items 335 A-C assigned to queue 320 N, etc.
- suspend/restore control unit 345 stores status information corresponding to the suspended state of the processor in memory 355 .
- memory 355 includes stored states 360 A-B, which are representative of any number of stored states that can be stored in memory 355 .
- suspend/restore control unit 345 stores the state of the processor in other locations.
- suspend/restore control unit 345 As suspend/restore control unit 345 is storing the state of the processor, work can continue to be executed and completed by execution units 350 A-N. However, no new work will be launched after the start of the suspension process. Finally, once suspend/restore control unit 345 finishes storing the state of the processor, then the suspension operation terminates, and the processor can shutdown, switch to executing another application, or perform another action.
- suspend/restore control unit 345 detects a request to restore one of the stored states 360 A-B. In response to detecting the request to restore one of the stored states 360 A-B, suspend/restore control unit 345 retrieves the work-item information from memory 355 and uses this information to restore work-items to queues 320 A-N. Suspend/restore control unit 345 also prevents threads 315 A-N and queues 320 A-N from launching new work-items while the restore operation is being implemented. Suspend/restore control unit 345 also retrieves information regarding the states of execution units 350 A-N and uses this information to restore the states of execution units 350 A-N to the state they were in at the time of the suspend operation. Once all of the work-items have been restored to their previous state, then suspend/restore control unit 345 allows threads 315 A-N and queues 320 A-N to start launching new work-items.
- N when displayed herein next to various structures is meant to generically indicate any number of elements for that structure (e.g., any number of queues 320 A-N). Additionally, different references within FIG. 3 that use the letter “N” (e.g., queues 320 A-N and threads 315 A-N) are not intended to indicate that equal numbers of the different elements are provided (e.g., the number of queues 320 A-N can differ from the number of threads 315 A-N).
- stored state 360 A includes the state of the processor (of FIG. 3 ) as shown in FIG. 3 .
- stored state 360 A includes information associated with each work-item which was stored in queues 320 A-C.
- this information includes work-item identifiers (IDs), queue IDs, queue entry IDs, execution unit IDs, status, and/or one or more other fields.
- the status information includes an indication on whether the work-item has launched yet.
- the status information can also include other information, such as the contents of one or more general purpose registers (GPRs), the contents of a local data share (LDS), a wave state, etc.
- GPRs general purpose registers
- LDS local data share
- the information shown in stored state 360 A is indicative of one embodiment. In other embodiments, other information can be included in the stored state 360 A which represents the state of the processor at the time when operation was suspended.
- FIG. 5 one embodiment of a method 500 for performing a suspend operation of in-flight work-items on a processor is shown.
- the steps in this embodiment and those of FIG. 6 are shown in sequential order. However, it is noted that in various embodiments of the described methods, one or more of the elements described are performed concurrently, in a different order than shown, or are omitted entirely. Other additional elements are also performed as desired. Any of the various systems, apparatuses, or computing devices described herein are configured to implement method 500 .
- a control unit detects a request to suspend a software application executing on a processor (block 505 ).
- the processor is a GPU.
- the processor is any of various other types of processors (e.g., DSP, FPGA, ASIC, multi-core processor).
- the control unit sends requests to a plurality of work creation units to stop creating new work (block 510 ).
- the work creation units are threads of an application, work queues of a GPU, and/or one or more other units.
- the control unit waits to receive acknowledgements from the plurality of work creation units that they have stopped creating new work (block 515 ).
- acknowledgements have been received from the plurality of work creation units to indicate that they have stopped creating new work (conditional block 520 , “yes” leg)
- the control unit initiates the suspend operation (block 525 ). If acknowledgements have not been received from the plurality of work creation units to indicate that they have stopped creating new work (conditional block 520 , “no” leg), then method 500 returns to block 515 .
- the control unit performs the suspend operation by determining which work-items are in-flight, determining which work-items have been assigned to which work creation units, and determining which execution units have been allocated for the in-flight work-items, (block 530 ).
- data associated with the in-flight work-items is streamed from various parts of the processor to the control unit, and embedded with this data is additional information specifying where the data came from within the processor.
- the control unit stores each piece of data and embedded tags associated with in-flight work-items to memory (block 535 ).
- the embedded tags include at least an ID to indicate which work creation unit the work-item was assigned and/or an ID to indicate on which execution unit each in-flight work-item was executing.
- the control unit describes each piece of data such that a later restore operation can restore the data back to the state that existed when the suspend operation was initiated.
- the saved data will be used to help the control unit redirect the saved data from memory back to the original source when instructed to do so by a restore operation. Then, the control unit sends a signal to software that the suspend operation has been completed (block 540 ). After block 540 , method 500 ends.
- a control unit receives a request to restore a previously suspended state of a processor (block 605 ). In response to receiving the request, the control unit clears the work creation units and causes the work creation units to be idle prior to starting a restore operation (block 610 ). Also, the control unit retrieves stored information about in-flight work items and corresponding execution units (block 615 ).
- control unit prevents the plurality of work creation units from launching new work-items (block 620 ).
- the control unit sends a request to each work creation unit to prevent the work creation unit from launching new work-items. Accordingly, the control unit prevents the plurality of work creation units from launching new work-items prior to restoring any in-flight work-items to each work creation unit. Then, the control unit restores in-flight work items to the same work creation units and execution units on which they were previously executing (block 625 ).
- condition block 630 If all of the in-flight work items have been restored to a work creation unit and/or execution unit (conditional block 630 , “yes” leg), then the control unit notifies the plurality of work creation units that new work-items can be launched (block 635 ). After block 635 , method 600 ends. If not all of the in-flight work items have been restored to a work creation unit and/or execution unit (conditional block 630 , “no” leg), then method 600 returns to block 625 .
- program instructions of a software application are used to implement the methods and/or mechanisms previously described.
- the program instructions describe the behavior of hardware in a high-level programming language, such as C.
- a hardware design language HDL
- the program instructions are stored on a non-transitory computer readable storage medium. Numerous types of storage media are available.
- the storage medium is accessible by a computing system during use to provide the program instructions and accompanying data to the computing system for program execution.
- the computing system includes at least one or more memories and one or more processors configured to execute program instructions.
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Advance Control (AREA)
Abstract
Description
- A graphics processing unit (GPU) is a complex integrated circuit that is configured to perform graphics-processing tasks. For example, a GPU can execute graphics-processing tasks required by an end-user application, such as a video-game application. GPUs are also increasingly being used to perform other tasks which are unrelated to graphics. The GPU can be a discrete device or can be included in the same device as another processor, such as a central processing unit (CPU).
- In many applications, such as graphics processing in a GPU, a sequence of work-items, which can also be referred to as threads, are processed so as to output a final result. In many modern parallel processors, for example, processors within a single instruction multiple data (SIMD) core synchronously execute a set of work-items. Typically, the synchronous executing of work-items are identical (i.e., have the identical code base). A plurality of identical synchronous work-items that are processed by separate processors are referred to as a wavefront or warp.
- During processing, one or more SIMD cores concurrently execute multiple wavefronts. Execution of the wavefront terminates when all work-items within the wavefront complete processing. Each wavefront includes multiple work-items that are processed in parallel, using the same set of instructions. Generally, the time required for each work-item to complete processing depends on a criterion determined by data within the work-item. As such, the work-items with the wavefront can complete processing at different times. When a user requests to suspend operation of the physical machine which includes the GPU, there are typically many in-flight operations executing on the GPU. Determining how to suspend operations on the GPU to enable a seamless restoration can be challenging. Accordingly, improved techniques for implementing suspend and restore operations on a processor are desired.
- The advantages of the methods and mechanisms described herein may be better understood by referring to the following description in conjunction with the accompanying drawings, in which:
-
FIG. 1 is a block diagram of one embodiment of a computing system. -
FIG. 2 is a block diagram of one embodiment of a GPU. -
FIG. 3 is a block diagram of one embodiment of a parallel processing environment. -
FIG. 4 illustrates one embodiment of a stored state. -
FIG. 5 is a generalized flow diagram illustrating one embodiment of a method for performing a suspend operation of in-flight work-items on a processor. -
FIG. 6 is a generalized flow diagram illustrating one embodiment of a method for performing a restore operation for a previously suspended processor state. - In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, one having ordinary skill in the art should recognize that the various embodiments may be practiced without these specific details. In some instances, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail to avoid obscuring the approaches described herein. It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements.
- Various systems, apparatuses, methods, and computer-readable mediums for suspending and restoring operations on a processor are disclosed. In one embodiment, a processor includes at least a plurality of execution units, a plurality of work creation units, and a control unit coupled to the plurality of execution units and the plurality of work creation units. In one embodiment, in response to detecting a request to suspend a software application executing on the processor, the control unit is configured to send requests to the plurality of work creation units to stop creating new work. The control unit waits until receiving acknowledgements in response to the requests from the plurality of work creation units prior to initiating a suspend operation. Once all work creation units have acknowledged that they will stop creating new work, the control unit initiates the suspend operation.
- In one embodiment, initiating the suspend operation involves determining which work-items are in-flight, determining which work creation units initiated the work-items, determining which execution units have been allocated for the in-flight work-items, and storing status information about the in-flight work items and corresponding execution units. The status information includes which work-items are in-flight, identifiers (IDs) of work creation units that initiated the work-items, IDs of the execution units that have been allocated for the in-flight work-items, and additional information. In another embodiment, initiating the suspend operation includes one or more other steps. In one embodiment, the work creation units are threads of the application and queues, and the execution units are single instruction, multiple data (SIMD) units.
- At a later point in time, the control unit receives a request to restore a previously suspended state of the processor. In one embodiment, in response to receiving the request, the control unit empties the work creation units and causes the work creation units to be idle prior to starting a restore operation. The control unit also retrieves stored information, corresponding to the previously suspended state, about in-flight work items and corresponding work creation units and execution units. Also, prior to restoring in-flight work-items to each work creation unit, the control unit prevents the plurality of work creation units from launching new work-items. Then, the control unit restores in-flight work items to each work creation unit and corresponding execution units. In one embodiment, each work-item is restored to a same work creation unit and/or a same execution unit on which the work-item was executing when the suspend operation occurred. Once the restoration of all in-flight work items has been completed to the work creation units and execution units, the control unit allows the plurality of work creation units to start launching new work-items.
- Referring now to
FIG. 1 , a block diagram of one embodiment of acomputing system 100 is shown. In one embodiment,computing system 100 includes system on chip (SoC) 105 coupled tomemory 150.SoC 105 can also be referred to as an integrated circuit (IC). In one embodiment, SoC 105 includesprocessing units 115A-N, input/output (I/O)interfaces 110, sharedcaches 120A-B,fabric 125, graphics processing unit (GPU) 130, and memory controller(s) 140.SoC 105 can also include other components not shown inFIG. 1 to avoid obscuring the figure.Processing units 115A-N are representative of any number and type of processing units. In one embodiment,processing units 115A-N are central processing unit (CPU) cores. In another embodiment, one or more ofprocessing units 115A-N are other types of processing units (e.g., application specific integrated circuit (ASIC), field programmable gate array (FPGA), digital signal processor (DSP)).Processing units 115A-N are coupled to sharedcaches 120A-B andfabric 125. - In one embodiment,
processing units 115A-N are configured to execute instructions of a particular instruction set architecture (ISA). Eachprocessing unit 115A-N includes one or more execution units, cache memories, schedulers, branch prediction circuits, and so forth. In one embodiment, theprocessing units 115A-N are configured to execute the main control software ofsystem 100, such as an operating system. Generally, software executed byprocessing units 115A-N during use can control the other components ofsystem 100 to realize the desired functionality ofsystem 100.Processing units 115A-N can also execute other software, such as application programs. -
GPU 130 includes at leastcontrol unit 132,queues 135, andcompute units 145A-N which are representative of any number and type of compute units that are used for graphics or general-purpose processing.GPU 130 is coupled to sharedcaches 120A-B andfabric 125. In one embodiment, GPU 130 is configured to execute graphics pipeline operations such as draw commands, pixel operations, geometric computations, and other operations for rendering an image to a display. In another embodiment,GPU 130 is configured to execute operations unrelated to graphics. In a further embodiment,GPU 130 is configured to execute both graphics operations and non-graphics related operations. - In one embodiment,
control unit 132 ofGPU 130 is configured to initiate a suspend operation in response to receiving a request to suspend an application executing onGPU 130. In one embodiment,control unit 132 causes the work creation units (e.g., queues 135) to stop launching new work-items prior to initiating the suspend operation. In one embodiment,control unit 132 starts a timer prior to initiating the suspend operation and sets a grace period for work creation units to become idle based on a number of clock cycles of the timer. In another embodiment,control unit 132 waits to receive acknowledgements from all work creation units that these units have stopped creating new work. Once all work creation units have been idled,control unit 132 initiates the suspend operation to determine and record information about the in-flight work-items, the work creation units that launched the in-flight work-items, the execution units which have been allocated for the in-flight work-items, and additional status information. In one embodiment,control unit 132 stores the information as storedstate 155 inmemory 150. In other embodiments,control unit 132 can allocate storedstate 155 in another location. - In one embodiment,
control unit 132 ofGPU 130 is configured to initiate a restore operation in response to receiving a request to restore a previously suspended application. In response to receiving the request,control unit 132 retrieves status information (e.g., stored state 155) corresponding to the previously suspended application. Also,control unit 132 empties and idles allqueues 135 and other work creation units. Then,control unit 132 restores the in-flight work-items specified in the status information to thesame queues 135 and execution units to which the work-items were previously allocated. After restoring all of the previously in-flight work-items,control unit 132 allows the work creation units to start launching new work-items. It is noted that a “restore operation” can also be referred to as a “resume operation”. - I/O interfaces 110 are coupled to
fabric 125, and I/O interfaces 110 are representative of any number and type of interfaces (e.g., peripheral component interconnect (PCI) bus, PCI-Extended (PCI-X), PCIE (PCI Express) bus, gigabit Ethernet (GBE) bus, universal serial bus (USB)). Various types of peripheral devices can be coupled to I/O interfaces 110. Such peripheral devices include (but are not limited to) displays, keyboards, mice, printers, scanners, joysticks or other types of game controllers, media recording devices, external storage devices, network interface cards, and so forth. -
SoC 105 is coupled tomemory 150, which includes one or more memory modules. Each of the memory modules includes one or more memory devices mounted thereon. In some embodiments,memory 150 includes one or more memory devices mounted on a motherboard or other carrier upon whichSoC 105 is also mounted. In one embodiment,memory 150 is used to implement a random access memory (RAM) for use withSoC 105 during operation. The RAM implemented can be static RAM (SRAM), dynamic RAM (DRAM), Resistive RAM (ReRAM), Phase Change RAM (PCRAM), or any other volatile or non-volatile RAM. The type of DRAM that is used to implementmemory 150 includes (but is not limited to) double data rate (DDR) DRAM, DDR2 DRAM, DDR3 DRAM, and so forth. Although not explicitly shown inFIG. 1 ,SoC 105 can also include one or more cache memories that are internal to theprocessing units 115A-N and/orcompute units 145A-N. In some embodiments,SoC 105 includes sharedcaches 120A-B that are utilized by processingunits 115A-N andcompute units 145A-N. In one embodiment,caches 120A-B are part of a cache subsystem including a cache controller. - In various embodiments,
computing system 100 can be a computer, laptop, mobile device, server or any of various other types of computing systems or devices. It is noted that the number of components ofcomputing system 100 and/orSoC 105 can vary from embodiment to embodiment. There can be more or fewer of each component/subcomponent than the number shown inFIG. 1 . For example, in another embodiment,SoC 105 can include multiple memory controllers coupled to multiple memories. It is also noted thatcomputing system 100 and/orSoC 105 can include other components not shown inFIG. 1 . Additionally, in other embodiments,computing system 100 andSoC 105 can be structured in other ways than shown inFIG. 1 . - Turning now to
FIG. 2 , a block diagram of one embodiment of a graphics processing unit (GPU) 200 is shown.GPU 200 includescommand processor 205, asynchronous compute engines (ACEs) 210A-N, computeunits 240,cache 245,memory controller 250, and direct memory access (DMA)engines GPU 200 also includes other logic and/or other components which are not shown inFIG. 2 to avoid obscuring the figure. It is also noted that the structure ofGPU 200 is merely one example of a structure which is used in one embodiment. In other embodiments,GPU 200 can be organized in other manners. - In one embodiment, the
command processor 205 also includes a save/restoremachine 212. In one embodiment, the save/restoremachine 212 is (or effectively implements) a finite state machine which is triggered and managed by firmware and/or microcode. When operations are suspended onGPU 200, status information associated with the in-flight work-items is saved to memory by save/restoremachine 212. Data associated with the in-flight work items is streamed from different units ofGPU 200 to save/restoremachine 212, and restore information identifying where the data came from is embedded alongside the data. For example, each piece of data sent to save/restoremachine 212 is tagged to indicate which unit generated the piece of data. In one embodiment, data is streamed over a common bus to save/restoremachine 212, and save/restoremachine 212 uses the embedded tags to determine which unit(s) generated each piece of data. Save/restoremachine 212 saves the status information and embedded tags to memory, and this stored data will be used later if a restore operation is requested. This restore information will be used by the save/restoremachine 212 to redirect the saved data in memory back to the original source in response to a restore operation being launched. - In one embodiment,
GPU 200 includes a plurality of queues for storing multiple work-items. For example,command processor 205 includesqueue 215,ACE 210A includesqueues 220, ACE 210B includesqueues 225, andACE 210N includesqueues 230.Queues -
ACEs 210A-N are representative of any number of asynchronous compute engines which are included inGPU 200. In one embodiment,ACEs 210A-N are configured to parse incoming commands and dispatch work to the processing elements incompute units 240. The processing elements incompute units 240 can also be referred to as “execution units”. In one embodiment,command processor 205 handles graphics tasks,ACEs 210A-N handle compute tasks, andDMA engines GPU 200 is able to dispatch work items without waiting for other tasks to complete, allowing independent command streams to be interleaved incompute units 240 and execute simultaneously. - Referring now to
FIG. 3 , a block diagram of one embodiment of aparallel processing environment 300. In one embodiment,parallel processing environment 300 includes a GPU. In other embodiments,parallel processing environment 300 includes other types of processors (e.g., digital signal processor (DSP), field programmable gate array (FPGA), application specific integrated circuit (ASIC), multi-core processor).Application 305 is representative of any number and type of software applications that are executing inparallel processing environment 300. In one embodiment,application 305 is a multi-threaded application, withapplication 305 includingthreads 315A-N, which are representative of any number of threads. - Suspend/restore
control unit 345 includes logic for suspending and restoring the state of the processor. In one embodiment, suspend/restorecontrol unit 345 includes a save/restore machine (e.g., save/restoremachine 212 ofFIG. 2 ). Depending on the embodiment, suspend/restorecontrol unit 345 can suspend and restore the state of the entire processor or suspend/restorecontrol unit 345 can suspend and restore the state of a portion of the processor. Suspend/restorecontrol unit 345 is implemented using any suitable combination of software and/or hardware, depending on the embodiment. In one embodiment, suspend/restorecontrol unit 345 will receive streamed data from various parts of the processor that manage work-items, and restore information will be embedded with the streamed data indicating where the work-items came from in the processor. Suspend/restorecontrol unit 345 will save this data to memory, and this saved data will be used later if a restore operation is requested. The saved data will be used by the suspend/restorecontrol unit 345 to redirect the saved data from memory back to the original source in response to a restore operation being launched. - When suspend/restore
control unit 345 detects a suspend request, suspend/restorecontrol unit 345 prevents new work from being generated by the work creation units (e.g.,threads 315A-N,queues 320A-N). In one embodiment, suspend/restorecontrol unit 345 generates a request forapplication 305 to stop sending new work toqueues 320A-N. In one embodiment, suspend/restorecontrol unit 345 also generates requests forqueues 320A-N to stop sending new work-items toexecution units 350A-N. Once suspend/restorecontrol unit 345 has received acknowledgments fromapplication 305 and/orqueues 320A-N that the launching of new work has been suspended, then suspend/restorecontrol unit 345 starts to store the status ofqueues 320A-N andexecution units 350A-N. The status includes information about work-items 325A-D assigned to queue 320A, work-items 330A-B assigned to queue 320B, work-items 335A-C assigned to queue 320N, etc. In one embodiment, suspend/restorecontrol unit 345 stores status information corresponding to the suspended state of the processor inmemory 355. For example,memory 355 includes storedstates 360A-B, which are representative of any number of stored states that can be stored inmemory 355. In other embodiments, suspend/restorecontrol unit 345 stores the state of the processor in other locations. - As suspend/restore
control unit 345 is storing the state of the processor, work can continue to be executed and completed byexecution units 350A-N. However, no new work will be launched after the start of the suspension process. Finally, once suspend/restorecontrol unit 345 finishes storing the state of the processor, then the suspension operation terminates, and the processor can shutdown, switch to executing another application, or perform another action. - At a later point of time, suspend/restore
control unit 345 detects a request to restore one of the stored states 360A-B. In response to detecting the request to restore one of the stored states 360A-B, suspend/restorecontrol unit 345 retrieves the work-item information frommemory 355 and uses this information to restore work-items toqueues 320A-N. Suspend/restorecontrol unit 345 also preventsthreads 315A-N andqueues 320A-N from launching new work-items while the restore operation is being implemented. Suspend/restorecontrol unit 345 also retrieves information regarding the states ofexecution units 350A-N and uses this information to restore the states ofexecution units 350A-N to the state they were in at the time of the suspend operation. Once all of the work-items have been restored to their previous state, then suspend/restorecontrol unit 345 allowsthreads 315A-N andqueues 320A-N to start launching new work-items. - It is noted that the letter “N” when displayed herein next to various structures is meant to generically indicate any number of elements for that structure (e.g., any number of
queues 320A-N). Additionally, different references withinFIG. 3 that use the letter “N” (e.g.,queues 320A-N andthreads 315A-N) are not intended to indicate that equal numbers of the different elements are provided (e.g., the number ofqueues 320A-N can differ from the number ofthreads 315A-N). - Turning now to
FIG. 4 , one embodiment of a storedstate 360A is shown. It can be assumed for the purposes of this discussion that storedstate 360A includes the state of the processor (ofFIG. 3 ) as shown inFIG. 3 . In one embodiment, storedstate 360A includes information associated with each work-item which was stored inqueues 320A-C. In one embodiment, this information includes work-item identifiers (IDs), queue IDs, queue entry IDs, execution unit IDs, status, and/or one or more other fields. In one embodiment, the status information includes an indication on whether the work-item has launched yet. In other embodiments, the status information can also include other information, such as the contents of one or more general purpose registers (GPRs), the contents of a local data share (LDS), a wave state, etc. It should be understood that the information shown in storedstate 360A is indicative of one embodiment. In other embodiments, other information can be included in the storedstate 360A which represents the state of the processor at the time when operation was suspended. - Referring now to
FIG. 5 , one embodiment of amethod 500 for performing a suspend operation of in-flight work-items on a processor is shown. For purposes of discussion, the steps in this embodiment and those ofFIG. 6 are shown in sequential order. However, it is noted that in various embodiments of the described methods, one or more of the elements described are performed concurrently, in a different order than shown, or are omitted entirely. Other additional elements are also performed as desired. Any of the various systems, apparatuses, or computing devices described herein are configured to implementmethod 500. - A control unit (e.g., suspend/restore
control unit 345 ofFIG. 3 ) detects a request to suspend a software application executing on a processor (block 505). In one embodiment, the processor is a GPU. In other embodiments, the processor is any of various other types of processors (e.g., DSP, FPGA, ASIC, multi-core processor). Next, in response to detecting the request, the control unit sends requests to a plurality of work creation units to stop creating new work (block 510). In one embodiment, the work creation units are threads of an application, work queues of a GPU, and/or one or more other units. Then, the control unit waits to receive acknowledgements from the plurality of work creation units that they have stopped creating new work (block 515). - If acknowledgements have been received from the plurality of work creation units to indicate that they have stopped creating new work (
conditional block 520, “yes” leg), then the control unit initiates the suspend operation (block 525). If acknowledgements have not been received from the plurality of work creation units to indicate that they have stopped creating new work (conditional block 520, “no” leg), thenmethod 500 returns to block 515. Afterblock 525, the control unit performs the suspend operation by determining which work-items are in-flight, determining which work-items have been assigned to which work creation units, and determining which execution units have been allocated for the in-flight work-items, (block 530). In one embodiment, data associated with the in-flight work-items is streamed from various parts of the processor to the control unit, and embedded with this data is additional information specifying where the data came from within the processor. Next, the control unit stores each piece of data and embedded tags associated with in-flight work-items to memory (block 535). In one embodiment, the embedded tags include at least an ID to indicate which work creation unit the work-item was assigned and/or an ID to indicate on which execution unit each in-flight work-item was executing. The control unit describes each piece of data such that a later restore operation can restore the data back to the state that existed when the suspend operation was initiated. The saved data will be used to help the control unit redirect the saved data from memory back to the original source when instructed to do so by a restore operation. Then, the control unit sends a signal to software that the suspend operation has been completed (block 540). Afterblock 540,method 500 ends. - Turning now to
FIG. 6 , one embodiment of amethod 600 for performing a restore operation for a previously suspended processor state is shown. A control unit receives a request to restore a previously suspended state of a processor (block 605). In response to receiving the request, the control unit clears the work creation units and causes the work creation units to be idle prior to starting a restore operation (block 610). Also, the control unit retrieves stored information about in-flight work items and corresponding execution units (block 615). - Next, the control unit prevents the plurality of work creation units from launching new work-items (block 620). In one embodiment, the control unit sends a request to each work creation unit to prevent the work creation unit from launching new work-items. Accordingly, the control unit prevents the plurality of work creation units from launching new work-items prior to restoring any in-flight work-items to each work creation unit. Then, the control unit restores in-flight work items to the same work creation units and execution units on which they were previously executing (block 625). If all of the in-flight work items have been restored to a work creation unit and/or execution unit (
conditional block 630, “yes” leg), then the control unit notifies the plurality of work creation units that new work-items can be launched (block 635). Afterblock 635,method 600 ends. If not all of the in-flight work items have been restored to a work creation unit and/or execution unit (conditional block 630, “no” leg), thenmethod 600 returns to block 625. - In various embodiments, program instructions of a software application are used to implement the methods and/or mechanisms previously described. The program instructions describe the behavior of hardware in a high-level programming language, such as C. Alternatively, a hardware design language (HDL) is used, such as Verilog. The program instructions are stored on a non-transitory computer readable storage medium. Numerous types of storage media are available. The storage medium is accessible by a computing system during use to provide the program instructions and accompanying data to the computing system for program execution. The computing system includes at least one or more memories and one or more processors configured to execute program instructions.
- It should be emphasized that the above-described embodiments are only non-limiting examples of implementations. Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/438,466 US10558489B2 (en) | 2017-02-21 | 2017-02-21 | Suspend and restore processor operations |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/438,466 US10558489B2 (en) | 2017-02-21 | 2017-02-21 | Suspend and restore processor operations |
Publications (2)
Publication Number | Publication Date |
---|---|
US20180239635A1 true US20180239635A1 (en) | 2018-08-23 |
US10558489B2 US10558489B2 (en) | 2020-02-11 |
Family
ID=63167162
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/438,466 Active 2037-09-22 US10558489B2 (en) | 2017-02-21 | 2017-02-21 | Suspend and restore processor operations |
Country Status (1)
Country | Link |
---|---|
US (1) | US10558489B2 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109766174A (en) * | 2018-12-24 | 2019-05-17 | 杭州数梦工场科技有限公司 | Resource regulating method, resource scheduling device and computer readable storage medium |
US11579876B2 (en) * | 2020-08-05 | 2023-02-14 | Advanced Micro Devices, Inc. | Reducing save restore latency for power control based on write signals |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11474861B1 (en) * | 2019-11-27 | 2022-10-18 | Meta Platforms Technologies, Llc | Methods and systems for managing asynchronous function calls |
Family Cites Families (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5555414A (en) | 1994-12-14 | 1996-09-10 | International Business Machines Corporation | Multiprocessing system including gating of host I/O and external enablement to guest enablement at polling intervals |
US5826084A (en) | 1997-03-25 | 1998-10-20 | Texas Instruments Incorporated | Microprocessor with circuits, systems, and methods for selectively bypassing external interrupts past the monitor program during virtual program operation |
US6397242B1 (en) | 1998-05-15 | 2002-05-28 | Vmware, Inc. | Virtualization system including a virtual machine monitor for a computer with a segmented architecture |
US7127548B2 (en) | 2002-04-16 | 2006-10-24 | Intel Corporation | Control register access virtualization performance improvement in the virtual-machine architecture |
US20040117532A1 (en) | 2002-12-11 | 2004-06-17 | Bennett Steven M. | Mechanism for controlling external interrupts in a virtual machine system |
US7454756B2 (en) | 2004-03-05 | 2008-11-18 | Intel Corporation | Method, apparatus and system for seamlessly sharing devices amongst virtual machines |
US7418584B1 (en) | 2004-05-11 | 2008-08-26 | Advanced Micro Devices, Inc. | Executing system management mode code as virtual machine guest |
US7545381B2 (en) * | 2005-11-10 | 2009-06-09 | Via Technologies, Inc. | Interruptible GPU and method for context saving and restoring |
US8108879B1 (en) * | 2006-10-27 | 2012-01-31 | Nvidia Corporation | Method and apparatus for context switching of multiple engines |
TWI328198B (en) * | 2006-12-11 | 2010-08-01 | Via Tech Inc | Gpu context switching system |
US20110219221A1 (en) * | 2010-03-03 | 2011-09-08 | Kevin Skadron | Dynamic warp subdivision for integrated branch and memory latency divergence tolerance |
US9652282B2 (en) * | 2011-11-08 | 2017-05-16 | Nvidia Corporation | Software-assisted instruction level execution preemption |
US10235208B2 (en) * | 2012-12-11 | 2019-03-19 | Nvidia Corporation | Technique for saving and restoring thread group operating state |
US9710874B2 (en) * | 2012-12-27 | 2017-07-18 | Nvidia Corporation | Mid-primitive graphics execution preemption |
US10360652B2 (en) * | 2014-06-13 | 2019-07-23 | Advanced Micro Devices, Inc. | Wavefront resource virtualization |
-
2017
- 2017-02-21 US US15/438,466 patent/US10558489B2/en active Active
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109766174A (en) * | 2018-12-24 | 2019-05-17 | 杭州数梦工场科技有限公司 | Resource regulating method, resource scheduling device and computer readable storage medium |
US11579876B2 (en) * | 2020-08-05 | 2023-02-14 | Advanced Micro Devices, Inc. | Reducing save restore latency for power control based on write signals |
Also Published As
Publication number | Publication date |
---|---|
US10558489B2 (en) | 2020-02-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11544106B2 (en) | Continuation analysis tasks for GPU task scheduling | |
US10552202B2 (en) | Software-assisted instruction level execution preemption | |
JP7087029B2 (en) | Improved functional callback mechanism between the central processing unit (CPU) and the auxiliary processor | |
US20130124838A1 (en) | Instruction level execution preemption | |
US11467870B2 (en) | VMID as a GPU task container for virtualization | |
CN110168497B (en) | Variable wavefront size | |
KR20170042217A (en) | Computer system comprising key-value device and method for providing interrupt handling | |
US10558489B2 (en) | Suspend and restore processor operations | |
US20130132711A1 (en) | Compute thread array granularity execution preemption | |
US10127076B1 (en) | Low latency thread context caching | |
US11074075B2 (en) | Wait instruction for preventing execution of one or more instructions until a load counter or store counter reaches a specified value | |
US10180789B2 (en) | Software control of state sets | |
JP2017167644A (en) | Data transfer device, data transfer method, and program | |
US8438335B2 (en) | Probe speculative address file |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: ADVANCED MICRO DEVICES, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MANTOR, MICHAEL J.;REEL/FRAME:041326/0571 Effective date: 20170217 Owner name: ADVANCED MICRO DEVICES, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ASHKAR, ALEXANDER FUAD;RAMSEY, RANDY WAYNE;MCCRARY, REX ELDON;AND OTHERS;SIGNING DATES FROM 20170216 TO 20170221;REEL/FRAME:041326/0534 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |