KR102746289B1 - Hybrid memory device and managing method therefor - Google Patents

Abstract

A method for managing a hybrid memory according to one embodiment of the present invention includes the steps of: receiving a plurality of first data to be cached from a first memory which is a nonvolatile storage class memory (SCM); collectively storing tag information of the plurality of first data in a first area within a cache line of a second memory which enables random access; and sequentially storing the plurality of first data in a second area within a cache line of the second memory.

Description

HYBRID MEMORY DEVICE AND MANAGING METHOD THEREFOR

The present invention relates to a high bandwidth memory (HBM) device and a management method thereof, and more particularly, to a high bandwidth memory device using heterogeneous memory, i.e., hybrid memory, for resolving performance degradation of HBM due to memory capacity limitations, and a management method thereof.

The material described in this section merely provides background information for the present embodiment and does not constitute prior art.

Memory devices are applied to many types of electronic devices, such as computers, mobile phones, PDAs, data loggers, game consoles, navigation devices, etc. Among the aforementioned electronic devices, various types of memory devices, such as NAND flash memory, NOR flash memory, SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), PCM (Phase Change Memory), etc. are applied. In accordance with the increase in operating speed and cache line size, memory devices can be packaged in packages within standardized specifications.

For example, a computing platform may have a structure that may be utilized as main memory, which may include a plurality of DRAM memories mounted in parallel. In this case, a read/write request addressed to the parallel memory modules within the memory device may be split across the parallel memory modules, such that each memory module may provide a subset of the total cache line request. The aforementioned memory devices typically have certain intrinsic parameters associated with, for example, read/write timing, memory page size, and/or addressing protocol.

High Bandwidth Memory (HBM) has been introduced for GPU applications that require large data such as deep learning. HBM is used in high-end GPUs and provides larger capacity and higher memory bandwidth than existing GDDR.

Demand paging can be used to run GPU applications that require large data in limited GPU memory. NVIDIA's Unified Memory is a representative example, which allows moving necessary pages between the CPU and GPU without much effort from the programmer.

To increase the total memory capacity of a GPU, methods are sometimes used to connect multiple GPUs with high-speed links, such as NVIDIA's NVLink.

The HBM used in existing high-end GPUs is smaller than the memory capacity required by important workloads such as deep learning and large-scale graph analysis, resulting in repeated transfers of data between the CPU and GPU, resulting in performance degradation.

This conventional demand paging has a problem that its performance is significantly degraded due to the long waiting time required for page fault processing in the host driver and the limited PCIe bandwidth. To solve this performance degradation problem, research is also being conducted on prefetch and eviction policies that utilize the regularity and locality of memory access patterns. However, in irregular workloads such as graph analysis, the existing prefetch and eviction policies have the problem of being inefficient because the locality is low and the access pattern is irregular.

Connecting multiple GPUs with high-speed links has the problem that the hardware cost of the GPU increases linearly as the total memory capacity increases due to additional costs caused by link interfaces and switches.

Korean Patent Registration No. KR 10-2231792 "Hybrid Memory Module and Its Operating Method" (Published on March 18, 2021)

The purpose of the present invention to solve the above problems is to propose a hybrid memory-based high-bandwidth memory device and management method with increased memory capacity without increasing the number of DRAM modules in the high-bandwidth memory.

The purpose of the present invention is to propose a hybrid memory-based high-bandwidth memory device and a management method that provide a memory capacity greater than that of a conventional high-bandwidth memory while providing data access performance equivalent to that of a conventional high-bandwidth memory composed only of DRAM modules.

An object of the present invention is to reduce performance degradation due to conventional memory over-subscription by utilizing a high bandwidth memory including heterogeneous stacked memory.

The purpose of the present invention is to propose a hybrid memory-based high-bandwidth memory device and management method that provide sufficient memory capacity even for GPU-based applications requiring large data.

The purpose of the present invention is to propose a hybrid memory-based high-bandwidth memory device and management method that provide improved performance by effectively controlling access frequency according to memory characteristics even in GPU-based applications requiring large data.

According to one embodiment of the present invention, a method for managing a hybrid memory includes: receiving a plurality of first data to be cached from a first memory, which is a nonvolatile storage class memory (SCM); collectively storing tag information of the plurality of first data in a first area within a cache line of a second memory capable of random access; and sequentially storing the plurality of first data in a second area within a cache line of the second memory.

A method for managing a hybrid memory according to one embodiment of the present invention may further include a step of storing error control information of a plurality of first data in a third area within a cache line of a second memory adjacent to the first area.

The first region may be a region of such a size that it can be read entirely in a single access to the second memory.

A method for managing a hybrid memory according to one embodiment of the present invention may further include a step of providing tag information on a plurality of first data stored in a first area to the host when information on a plurality of first data is requested from the host.

A method for managing a hybrid memory according to one embodiment of the present invention may further include a step in which a host stores tag information for a plurality of first data stored in a first area in a part of a level 2 cache as identification information for the plurality of first data.

The first memory and the second memory can share a common address of a first number of bits, and the tag information of the plurality of first data can include offset information of a second number of bits between a first address on the first memory where the plurality of first data are stored, excluding the common address, and a second address on the second memory.

A method for managing a hybrid memory according to one embodiment of the present invention includes: calculating a device sensitivity between a first memory and a second memory for a first request when data corresponding to a first request of a host is not stored in a second memory capable of random access as data stored in a non-volatile first memory; and determining whether to bypass the second memory and directly access the first memory for the first request based on the device sensitivity. The device sensitivity is calculated based on a first cost of the first memory according to a type of access of the first request, and a second cost of the second memory for processing the first request.

In the step of calculating the device sensitivity, the device sensitivity can be calculated by dividing it by the number of consecutive data included in the first request.

In the step of calculating the device sensitivity, the device sensitivity can be discretized to any value between a predetermined minimum value and an observed maximum value.

A method for managing a hybrid memory according to one embodiment of the present invention may further include: calculating an affinity of a first request to a second memory if it is determined not to bypass the second memory based on device sensitivity; and determining whether to replace a portion of a cache line of the second memory by the first request based on a comparison result between the affinity of the first request to the second memory and the affinity of the victim data to be expelled from the second memory to the second memory.

In the step of calculating the affinity of the first request to the second memory, the affinity to the second memory can be determined based on the access frequency to data corresponding to the first request and the device sensitivity of the first request.

In the step of calculating the affinity of the first request to the second memory, the affinity to the second memory can be discretized to any value between a predetermined minimum value and an observed maximum value.

A method for managing a hybrid memory according to one embodiment of the present invention may further include a step of bypassing the second memory and directly accessing the first memory for the first request to process the first request when it is determined not to replace a part of a cache line of the second memory in response to the first request.

A method for managing a hybrid memory according to one embodiment of the present invention may further include a step of reducing the affinity of victim data for the second memory according to a predetermined condition in response to an event in which data corresponding to the first request is not stored in the second memory, when it is determined not to replace a part of a cache line of the second memory according to the first request.

A hybrid memory device according to one embodiment of the present invention includes a first memory which is a nonvolatile storage class memory (SCM); a second memory which is randomly accessible; and a memory controller. The memory controller collectively stores tag information of a plurality of first data to be cached from the first memory in a first area within a cache line of the second memory, and sequentially stores the plurality of first data in a second area within a cache line of the second memory.

The memory controller can store error control information of a plurality of first data in a third area within a cache line of a second memory adjacent to the first area.

The first region may be a region of such a size that it can be read entirely in a single access to the second memory.

The memory controller can provide tag information on a plurality of first data stored in a first area to the host when information on a plurality of first data is requested from the host.

The memory controller can provide tag information for a plurality of first data to the host so that the host can store tag information for a plurality of first data stored in a first area in some area of a level 2 cache as identification information for the plurality of first data.

The first memory and the second memory can share a common address of a first number of bits. The tag information of the plurality of first data can include offset information of a second number of bits between a first address on the first memory where the plurality of first data are stored, excluding the common address, and a second address on the second memory.

According to an embodiment of the present invention, by utilizing heterogeneous memory within a high bandwidth memory, memory capacity can be increased without increasing the number of DRAM modules, and data access performance at a level equivalent to that of a memory composed solely of DRAM modules can be provided.

According to an embodiment of the present invention, performance degradation due to existing memory over-subscription can be reduced by using a high-bandwidth memory including a heterogeneous memory stack (HMS).

According to an embodiment of the present invention, performance can be improved by providing sufficient memory capacity even in GPU applications that require large data, such as deep learning and large-scale graph analysis.

According to an embodiment of the present invention, performance can be improved even in GPU applications requiring large data by reducing access to memory with high access cost (slow memory).

FIG. 1 is a conceptual diagram illustrating a hybrid memory-based high-bandwidth memory device according to one embodiment of the present invention.
FIG. 2 is a conceptual diagram illustrating a hybrid memory device and its peripheral devices according to one embodiment of the present invention.
FIG. 3 is a flowchart illustrating a management method of a hybrid memory device according to one embodiment of the present invention.
FIG. 4 is a conceptual diagram illustrating a CTC concept, which is a method for storing tags, in relation to a hybrid memory device according to one embodiment of the present invention.
FIG. 5 is a conceptual diagram illustrating an ATIC structure, which is a method for storing tags in a hybrid memory device according to one embodiment of the present invention.
FIG. 6 is a flowchart illustrating a management method of a hybrid memory device according to one embodiment of the present invention.
Figure 7 is a flowchart illustrating a portion of the method of Figure 6 in more detail.
FIG. 8 is a conceptual diagram illustrating a process for calculating device affinity in a hybrid memory device according to one embodiment of the present invention.
FIG. 9 is a conceptual diagram illustrating an example of a memory controller, memory management device, or computing system within a generalized hybrid memory device capable of performing at least a portion of the processes of FIGS. 1 through 8.

The present invention can have various modifications and various embodiments, and specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to specific embodiments, but should be understood to include all modifications, equivalents, or substitutes included in the spirit and technical scope of the present invention.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are only used to distinguish one component from another. For example, without departing from the scope of the present invention, the first component could be referred to as the second component, and similarly, the second component could also be referred to as the first component. The term "and/or" includes any combination of a plurality of related listed items or any item among a plurality of related listed items.

In the embodiments of the present application, “at least one of A and B” can mean “at least one of A or B” or “at least one of combinations of one or more of A and B.” Furthermore, in the embodiments of the present application, “at least one of A and B” can mean “at least one of A or B” or “at least one of combinations of one or more of A and B.”

When it is said that a component is "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. On the other hand, when it is said that a component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprises" or "has" and the like are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms defined in commonly used dictionaries, such as those defined in common dictionaries, should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense unless expressly defined in this application.

Meanwhile, even if it is a technology known prior to the filing date of this application, it may be included as a part of the composition of the invention of this application if necessary, and this will be described in this specification within the scope that does not obscure the purpose of the invention. However, in explaining the composition of the invention of this application, a detailed description of matters that were known prior to the filing date of this application and could be clearly understood by those skilled in the art may obscure the purpose of the invention, and therefore, an excessively detailed description of the known technology will be omitted.

For example, a technology for stacking DRAM and storage-class memory (SCM) and using TSV (Through silicon via) to connect corresponding terminals of DRAM and SCM can utilize known technologies prior to the filing of the present invention, and at least some of these known technologies can be applied as element technologies necessary for implementing the present invention.

However, the purpose of the present invention is not to claim rights to these known technologies, and the contents of the known technologies may be included as part of the present invention within a scope that does not deviate from the purpose of the present invention.

Hereinafter, with reference to the attached drawings, a preferred embodiment of the present invention will be described in more detail. In order to facilitate an overall understanding in describing the present invention, the same reference numerals are used for the same components in the drawings, and redundant descriptions of the same components are omitted.

FIG. 1 is a conceptual diagram illustrating a hybrid memory-based high-bandwidth memory device according to one embodiment of the present invention.

In one embodiment of the present invention illustrated in FIG. 1, a high bandwidth memory (130) (HBM, High Bandwidth Memory) based on a heterogeneous stacked memory (HMS, Heterogeneous Memory Stack) can be configured by integrating a storage class memory (SCM) stack corresponding to a first memory (110) and a DRAM stack corresponding to a second memory (120) into a single chip through 3D stacking.

Communication and interaction with external elements including the HBM (130) and the GPU (200) of FIG. 1 can follow the basic configuration of existing high-bandwidth memory. Referring to FIG. 1, the HBM (130) of FIG. 1 can be implemented by replacing the DRAM dies corresponding to the upper ranks in the existing HBM with SCM dies.

In order to alleviate performance degradation due to SCM in the heterogeneous stacked memory-based HBM (130) of Fig. 1, it can be understood that the DRAM corresponding to the second memory (120) operates as a cache of the SCM with respect to the SCM corresponding to the second memory (120). Since the SCM generally has a higher memory density than the DRAM, the heterogeneous stacked memory-based HBM (130) illustrated in Fig. 1 can provide a larger memory capacity than the existing HBM.

In general, HBM is a technology that aims to reduce the heat and energy consumption generated when data is transmitted through serial communication in the existing DDRx standard, and to reduce data transmission time and energy consumption by transmitting high-bandwidth data at once through an intensive high-bandwidth bus.

In general, HBM is much faster than conventional DDRx-based memory technology, consumes less power, and takes up less space. It is also particularly popular in high-performance computing (HPC) and artificial intelligence (AI) applications that require a lot of resources. However, there are problems that limit its utilization due to its high price, thermal management issues, and the fact that applications sometimes need to be modified.

In general, HBM's performance is improved through efforts to increase bandwidth by widening the bus width and shortening the physical time required to move data by vertically stacking using the TSV structure.

According to one embodiment of the present invention, it can be used as a GPU memory when executing a GPU application that requires large data, such as deep learning or large-scale graph analysis.

Specifically, according to one embodiment of the present invention, when executing a GPU application requiring large data such as deep learning or large-scale graph analysis, a heterogeneous memory stack (HMS) is used as the memory of the GPU (200), thereby reducing performance degradation due to memory oversubscription compared to the prior art.

FIG. 2 is a conceptual diagram illustrating the interaction between a hybrid memory device (100) and its peripheral devices according to one embodiment of the present invention.

The application layer (300) can control the GPU (200) to allocate a given task to the GPU (200) so that the GPU (200) can perform the assigned task.

The application layer (300) may include a separate processor. The GPU (200) may include at least one processor separately from the application layer (300).

A system memory/storage (320) corresponding to a processor of an application layer (300) may be provided. A memory controller (140) according to an embodiment of the present invention may store data read from the system memory/storage (320) in a first memory (110) or a second memory (120) within a high bandwidth memory (130). At this time, the memory controller (140) may also partially perform a role similar to a type of communication interface.

A hybrid memory device (100) according to one embodiment of the present invention includes a first memory (110) which is a nonvolatile storage class memory (SCM); a second memory (120) which enables random access; and a memory controller (140). The memory controller (140) collectively stores tag information of a plurality of first data to be cached from the first memory (110) in a first area within a cache line of the second memory (120), and sequentially stores the plurality of first data in a second area within a cache line of the second memory (120).

High bandwidth memory (HBM) (130) may refer to a heterogeneous stacked memory including a first memory (110) and a second memory (120). A structure including the HBM (130) and the memory controller (140) may be called a hybrid memory device (100). However, this is only one embodiment of the present invention, and in another embodiment of the present invention, the memory controller (140) may be included in the hybrid memory device (100) or may be arranged between the hybrid memory device (100) and the GPU (200) to function as a memory management device of the hybrid memory device (100).

The memory controller (140) can store error control information of a plurality of first data in a third area within a cache line of a second memory (120) adjacent to the first area. The error control information can include an error control code (ECC). In some embodiments, ECC is also called Error Correction Code.

The memory controller (140) can provide tag information on a plurality of first data stored in a first area to the host when information on a plurality of first data is requested from the host.

The memory controller (140) can provide tag information for a plurality of first data to the host so that the host can store tag information for a plurality of first data stored in a first area in some area of a level 2 cache (220) as identification information for the plurality of first data.

The L2 cache (200) is typically placed within the GPU (200) or may be placed very close to the GPU (200). Therefore, the cost of accessing the L2 cache (220) from the GPU (200) side may be significantly smaller than the cost of accessing the hybrid memory device (100).

Memory-to-data operations for data centers, servers, and artificial neural network operations, which HBM typically targets, are continually demanding higher bandwidth and larger memory capacities.

In general, most of the HBMs of the prior art are configured based on DRAM. However, assuming that regular access is frequently possible due to the characteristics of memory-data operations, one embodiment of the present invention proposes a method of replacing a portion of the DRAM in the HBM (130) structure of FIG. 1 with a storage-class memory (SCM) in order to increase memory capacity.

At this time, SCM has disadvantages such as slow response speed and large write cost while being more economical than DRAM, so in order to improve overall performance, it is necessary to effectively manage data storage/access between DRAM part and SCM part so that SCM operation is not revealed.

The present invention is characterized by effectively managing data storage/access between a DRAM part and an SCM part as an HBM (130) in a hybrid memory device (100) based on a heterogeneous stacked memory (HMC) in which DRAM-SCM are combined.

Generally, a hybrid memory module refers to a memory module that includes both volatile memory (e.g., dynamic random-access memory (DRAM)) and non-volatile memory (e.g., flash memory) as the primary data storage devices. An example of a hybrid memory module is a hybrid dual in-line memory module (DIMM) that integrates DRAM and flash memory. In a typical configuration, DRAM can be used as a cache memory for data stored in flash memory. To quickly access the DRAM cache, metadata of the DRAM cache can be stored in the static random-access memory (SRAM) of the hybrid memory module. In this case, the SRAM can function as a cache at a higher level (closer to the CPU) than DRAM.

However, the storage space required for metadata in the DRAM cache may be larger than the available storage space of the SRAM. The memory capacity of the SRAM integrated in the hybrid DIMM may be kept relatively small due to its price. Due to the limited storage space size of the SRAM, the entire metadata of the DRAM cache cannot fit into the SRAM, and as a result, the remaining part of the metadata that cannot fit into the SRAM must be stored in the DRAM. In this case, the slow access speed to the metadata stored in the DRAM may cause a performance degradation when accessing the data.

To address these issues, several approaches have been proposed. The first approach is to reduce the size of the metadata stored in the SRAM. For example, the metadata size can be reduced by reducing the number of cache lines stored in the SRAM or by reducing the size of the cache lines. However, in these cases, the reduced cache line size and/or the reduced number of cache lines stored at one time may negatively impact the hit rate, and multiple page reads from the flash memory may be required in case of a cache miss. In another example, the cache associativity can be reduced by reducing the tag bits and replacement bits, but this approach may also negatively impact the hit rate. In another example, the replacement policy can be replaced without utilizing the replacement bits.

Even in these prior art examples, the test results show that the combination of these effects to reduce metadata size only reduces a portion of the required metadata size. Therefore, the problem of limited size of SRAM for storing metadata may continue as the data storage capacity of flash memory and the size of DRAM cache increase.

The present invention proposes a method for reducing the frequency of cache probes without reducing the size of a target of a single DRAM cache probe as a solution that is differentiated from such conventional technologies. That is, the present invention proposes a hybrid memory and its management method that improves the overall memory access performance by reducing the frequency of cache probes without reducing the size of a cacheline or the number of cachelines stored in an L2 cache (220).

The hybrid memory and management method according to one embodiment of the present invention can be applied even when the capacity of the SCM and the size of the DRAM cache increase, and can still reduce the frequency of cache probes and improve the overall memory access performance.

A hybrid memory device (100) including an HBM (130) according to one embodiment of the present invention can effectively provide DRAM-like performance to a GPU (200) based on characteristic parameters of a DRAM and an SCM included in a heterogeneous stacked memory. At this time, the hybrid memory device (100) and management method according to one embodiment of the present invention can minimize the degradation of memory access performance due to the SCM as viewed from the GPU (200) side by using a DRAM cache bypass mechanism and provide overall memory access performance, such as memory access bandwidth, similar to that of a conventional HBM composed only of DRAM.

Unlike studies on bypass of CPU-optimized DRAM caches that include only conventional DRAMs, the hybrid memory device (100) including HBM (130) of the present invention can efficiently use memory bandwidth by using a DRAM cache bypass mechanism that reflects the memory access pattern of GPU workload and the characteristics of SCM such as long write time. The DRAM cache bypass mechanism of the hybrid memory device (100) of the present invention may use a device-sensitivity score that reflects the spatial locality of memory access and the access type (read or write) and a DRAM-affinity score value that multiplies the access frequency of a page by the device-sensitivity score.

The DRAM cache bypass mechanism of the hybrid memory device (100) according to one embodiment of the present invention may be understood as an SCM-aware DRAM cache bypass policy. Whether to bypass the DRAM cache may be determined by multidimensionally considering the performance difference between the DRAM and the SCM, the type of cache-missed access, and the overall size of the cache-missed access.

The DRAM cache bypass mechanism of the hybrid memory device (100) according to one embodiment of the present invention is executed when a DRAM cache miss occurs, and a first step may be performed in which a device-sensitivity score is calculated for a cache-missed access and compared with an average device-sensitivity score managed for each channel of the memory. The comparison process may be performed after discretizing each score. The second step may be performed only if the cache-missed access is greater than the channel average, and otherwise the DRAM cache may be bypassed immediately. In the second step, the DRAM-affinity score may be calculated by multiplying the device-sensitivity score of the cache-missed access by the access frequency of the corresponding page, and the discretized DRAM-affinity score may be compared with the discretized DRAM-affinity score of a victim candidate. DRAM cache line replacement may occur only if the value of the cache-missed access is greater than that of the victim candidate, and otherwise the DRAM cache may be bypassed.

The specific configuration of the hybrid memory and management method of the present invention is described in the embodiments below in FIG. 3.

FIG. 3 is a flowchart illustrating a management method of a hybrid memory device (100) according to one embodiment of the present invention.

A method for managing a hybrid memory according to one embodiment of the present invention includes a step (S410) of receiving a plurality of first data to be cached from a first memory (110) which is a non-volatile storage class memory (SCM); a step (S420) of collectively storing tag information of the plurality of first data in a first area within a cache line of a second memory (120) which enables random access; and a step (S430) of sequentially storing the plurality of first data in a second area within a cache line of the second memory (120).

A method for managing a hybrid memory according to one embodiment of the present invention may further include a step (S440) of storing error control information of a plurality of first data in a third area within a cache line of a second memory (120) adjacent to the first area.

The first area may be an area of a size that can be read entirely with a single access to the second memory (120).

A method for managing a hybrid memory according to one embodiment of the present invention may further include a step of providing tag information on a plurality of first data stored in a first area to the host when information on a plurality of first data is requested from the host.

A method for managing a hybrid memory according to one embodiment of the present invention may further include a step in which a host stores tag information for a plurality of first data stored in a first area in a part of a level 2 cache as identification information for the plurality of first data.

The first memory (110) and the second memory (120) can share a common address of a first number of bits, and the tag information of the plurality of first data can include offset information of a second number of bits between a first address excluding the common address among the first addresses on the first memory (110) where the plurality of first data are stored and a second address on the second memory (120).

FIG. 4 is a conceptual diagram illustrating the CTC concept, which is a method for storing tags, in relation to a hybrid memory device (100) according to one embodiment of the present invention.

In a hybrid memory device (100) according to one embodiment of the present invention, meta-data information tagging can be stored in an L2 cache (220) (typically SRAM).

In a hybrid memory device (100) according to one embodiment of the present invention, a CTC (configurable Tag Cache) concept that stores DRAM cache tags in a part of the L2 cache (220) space may be proposed to reduce traffic caused by DRAM cache probes.

In order to reduce excessive DRAM cache probe traffic that occurs even when using the DRAM cache bypass mechanism described in the embodiment of FIG. 2 and through the embodiments of FIGS. 6 to 8 described below, a Configurable Tag Cache (CTC) that stores tags of a DRAM cache on-chip at a low cost may be used in one embodiment of the present invention. To this end, in one embodiment of the present invention, some ways of the L2 cache (220) of the GPU (200) may be used as the CTC, and the ways used may be flexibly adjusted.

The CTC of FIG. 4 can be implemented by utilizing the cache residency control function of the L2 cache (220) of the GPU (200). The generally known L2 cache residency control function is a function that provides management authority for data to be stored or controlled in the cache. However, the CTC according to one embodiment of the present invention can be differentiated from the conventional technology in that it collectively stores tags for cache probe for the DRAM cache.

Referring to Fig. 4, only data for the original operation of the L2 cache can be stored in some areas (222) of the L2 cache (220).

In the remaining part of the L2 cache (220) (224), a CTC area that can store necessary tags as a Tags cache in addition to the original operation of the L2 cache can be set. At this time, the size of the area (224) can be adjusted as needed.

For example, in area (222), a space divided into 128 bytes in size may be set for the original operation of the L2 cache, and in area (224), a space divided into 32 bytes in size may be set for storing tags.

FIG. 5 is a conceptual diagram illustrating an ATIC structure, which is a method for storing tags in a hybrid memory device (100) according to one embodiment of the present invention.

According to the embodiment illustrated in FIG. 5, an ATIC (Aggregated Tags-Inside-Cacheline) structure is illustrated, which is a method in which tags are stored within an L2 cache (220).

In a memory structure that borrows the CTC concept of FIG. 4, when a cache miss occurs and a DRAM cache probe occurs, an embodiment of the present invention can manage DRAM cache tags using an Aggregated Tags-Inside-Cacheline (ATIC) structure to reduce the traffic that occurs at this time. In the ATIC structure of an embodiment of the present invention, DRAM cache tags corresponding to one DRAM row can be stored collectively in the ECC part of the first single 32B column. Through this method, an embodiment of the present invention can obtain tag information of DRAM cache lines corresponding to one DRAM row with one DRAM access. An embodiment of the present invention can store and manage ECC information of the first single 32B column data in the spare ECC parts of the following columns belonging to the same cache line.

Referring to FIG. 5, by intensively/collectively storing DRAM cache tags corresponding to a plurality of first data stored in one row within a DRAM-based second memory (120) in the first area (510), which is the ECC portion of the first single 32-byte column, the traffic occurring during a DRAM cache probe can be reduced.

A memory controller (140) of a hybrid memory device (100) according to one embodiment of the present invention collectively stores tag information of a plurality of first data to be cached from a first memory (110) in a first area (510) within a cache line of a second memory (120), and sequentially stores the plurality of first data in a second area (520) within a cache line of the second memory (120).

The memory controller (140) can store error control information of a plurality of first data in a third area (530) within a cache line of a second memory (120) adjacent to the first area (510). The error control information can include an error control code (ECC). In some embodiments, ECC is also called Error Correction Code.

The first area (510) may be an area of a size that can be read entirely with a single access to the second memory (120). The size that can be read with a single access to the second memory (120) may be, for example, one byte or one of a 16/32/64/128-bit word.

For example, tags of a plurality of first data stored in one row of the second memory (120) may be generated as collective tag information of the size of the first area (510), and the collective tag information may be stored in the first area (510). The collective tag information stored in the first area (510) may be transferred from the second memory (120) to the L2 cache (220) as tag information for checking whether the plurality of first data are stored in the second memory (120) or the first memory (110). In a cache probe operation for transferring such tag information, according to an embodiment of the present invention, the collective tag information, that is, according to the ATIC structure, a DRAM cache probe for the plurality of first data may be completed with only one access to the first area (510) in the ATIC.

That is, while in the prior art, multiple cache probes are performed when performing cache probes on multiple first data, in the embodiment of the present invention, cache probes on multiple first data can be completed by a single access to the first area (510), thereby reducing the number of memory accesses and lowering memory access costs.

As described above, the embodiment of FIG. 4 can reduce the number of DRAM accesses for cache probes. In addition, the embodiment of FIG. 5 can load all tags of DRAM cache lines corresponding to a DRAM row into the SRAM cache with a single DRAM access regardless of the location of the SRAM cache, thereby also reducing the number of DRAM accesses for cache probes.

At this time, by combining the embodiment of Fig. 5 and the embodiment of Fig. 4, the overall memory access cost (including both SRAM and DRAM) for cache probe can be further reduced.

In the case of using DRAM as a cache for SCM in heterogeneous stacked memory-based HBM (130), since the target is GPU (200) applications, unlike the general DRAM cache technology that uses cache lines of 64-128B, a larger cache line of 256B can be used to utilize the high spatial locality of GPU workload and alleviate the high activation cost of SCM.

The ATIC structure of the present invention can provide an effective cache probe reduction effect even for a large cache line for a GPU (200).

Meanwhile, the configurations of FIGS. 4 and 5 can further increase the effect of reducing tag probe access to DRAM when assuming an NVRAM-based SCM.

NVRAM-based SCM can mean a type of memory that enables at least row-based random access, in-place writing, and does not require a separate erase operation, such as phase change memory (PCM).

For example, the first memory (110) and the second memory (120) may share a common address of a first number of bits. The SCM and DRAM may adopt a direct-mapped cache organization.

Tag information of a plurality of first data may include offset information of a second number of bits between a first address excluding a common address among first addresses on a first memory (110) where a plurality of first data are stored and a second address on a second memory (120).

For example, if the SCM is n times the size of the DRAM, the address mapping between the SCM area and the DRAM area can be implemented as 1:n. At this time, among the first addresses on the first memory (110), there is a common address field that is shared in common with the second address of the second memory (120), and an address field other than the second address can be implemented as offset information of the first address of the first memory (110). If the common address is configured as the first number of bits, the offset information can be implemented as the second number of bits.

The memory controller (140) can manage only the offset information excluding the common address part of the data requested from the host as cache tags information, so when combined with the ATIC structure and the CTC concept, a larger number of first data can be managed through a single DRAM cache probe, so it has an advantage in applications requiring large data, and the frequency of DRAM cache probes can be reduced.

Although one embodiment of the present invention adopts a direct-mapped cache organization, the spirit of the present invention is not limited thereto, and other embodiments of the present invention may be modified to achieve the intended purpose by effectively managing addresses even in a cell-associated cache organization.

Another conventional technique that can be compared with an embodiment of the present invention that combines the embodiments of FIGS. 4 and 5 may be a method of compressing and storing tags for a cache probe. However, the method of compressing and storing tags is inefficient because it requires computing resources to compress and store or decompress tags, and also requires separate cache level hardware resources for compressing/decompressing tags, whereas the collective tag information according to an embodiment of the present invention is more efficient than the conventional technique and can significantly reduce memory access costs because it does not require separate computing resources or hardware resources for compression/decompression.

FIG. 6 is a flowchart illustrating a management method of a hybrid memory device (100) according to one embodiment of the present invention.

Figure 7 is a flowchart illustrating a portion of the method of Figure 6 in more detail.

A method for managing a hybrid memory according to one embodiment of the present invention can be performed by a memory controller (140).

A method for managing a hybrid memory according to one embodiment of the present invention includes the steps of: calculating a device sensitivity between a first memory (110) and a second memory (120) for the first request when data corresponding to a first request of a host, which is stored in a non-volatile first memory (110), is not stored in a second memory (120) that allows random access (S610); and determining whether to bypass the second memory (120) and directly access the first memory (110) for the first request based on the device sensitivity (S630). The device sensitivity is calculated based on a first cost of the first memory (110) according to the type of access of the first request, and a second cost of the second memory (120) for processing the first request.

In the step of calculating device sensitivity (S620), the device sensitivity can be calculated by dividing it by the number of consecutive data included in the first request.

In the step of calculating the device sensitivity (S620), the device sensitivity can be discretized into any value between a predetermined minimum value and an observed maximum value.

A method for managing a hybrid memory according to one embodiment of the present invention may further include a step (S640) of calculating an affinity of a first request to a second memory (120) when it is not determined to bypass the second memory (120) based on device sensitivity (S630: Yes); and a step (S650) of determining whether to replace a part of a cache line of the second memory (120) by the first request based on a comparison result between the affinity of the first request to the second memory (120) and the affinity of the big team data to be expelled from the second memory (120) to the second memory (120).

In the step (S640) of calculating the affinity for the second memory (120) of the first request, the affinity for the second memory (120) can be determined based on the access frequency for data corresponding to the first request and the device sensitivity of the first request.

In the step (S640) of calculating the affinity for the second memory (120) of the first request, the affinity for the second memory (120) can be discretized into any value between a predetermined minimum value and an observed maximum value.

A hybrid memory management method according to one embodiment of the present invention may further include a step (S680) of bypassing the second memory (120) and directly accessing the first memory (110) for processing the first request when it is not determined to replace a part of the cache line of the second memory (120) by the first request (S650: No).

A hybrid memory management method according to one embodiment of the present invention may further include a step (S670) of reducing the affinity of the big data for the second memory (120) according to a predetermined condition in response to an event in which data corresponding to the first request is not stored in the second memory (120) when it is not determined to replace a part of the cache line of the second memory (120) by the first request (S650: No).

Unlike studies on bypass of CPU-optimized DRAM caches that include only conventional DRAMs, the hybrid memory device (100) including HBM (130) of the present invention can efficiently use memory bandwidth by using a DRAM cache bypass mechanism that reflects the memory access pattern of GPU workload and the characteristics of SCM such as long write time. The DRAM cache bypass mechanism of the hybrid memory device (100) of the present invention may use a device-sensitivity score that reflects the spatial locality of memory access and the access type (read or write) and a DRAM-affinity score value that multiplies the access frequency of a page by the device-sensitivity score.

According to an embodiment of the present invention, a DRAM cache bypass mechanism of a hybrid memory device (100) is executed when a DRAM cache miss occurs (S610), and a first step (S630) may be performed in which a device-sensitivity score is calculated for a cache-missed access (S620) and compared with an average device-sensitivity score managed for each channel of a memory. The comparison process may be performed after discretizing each score. Only when the cache-missed access is greater than the channel average value, the second step (S640 to S650) may be performed, and otherwise, the DRAM cache may be bypassed immediately (S680). In the second step (S640~S650), the DRAM-affinity score is calculated by multiplying the device-sensitivity score of the cache-missed access by the access frequency of the corresponding page (S640), and the discretized DRAM-affinity score can be compared with the discretized DRAM-affinity score of the victim candidate (S650). Only if the value of the cache-missed access is greater than that of the victim candidate (S650: Yes), DRAM cache line replacement occurs (S660), otherwise the DRAM cache can be bypassed (S680).

A cache is a memory for temporary storage of data in a data processing device. Typically, a cache is a small, high-speed memory that stores copies of data that constitute a subset from a backing storage device. The backing storage device is typically a large, low-speed memory or data storage device. The data in the cache is used by cache clients, such as a central processing unit (CPU). The performance of the CPU is enhanced when frequently used data is available in the cache, thereby avoiding the latency associated with reading data from the backing storage device. Each entry in the cache contains the data itself, a valid bit, and optionally one or more status bits, along with a tag relating to the location of the original data in the backing storage device. The size of the cache is determined by the need to store the tags and the valid bit and status bits in addition to the data itself.

In applications that require caching large amounts of data, large caches may be provided. For example, large tag storage may be required for large off-chip caches outside the CPU. For example, tag storage requirements for large off-chip DRAM caches often exceed what can actually be stored in on-chip SRAM (e.g., Level 2 (L2) Cache (220)). However, storing tags in DRAM may incur a very large latency penalty. If both tags and data are read from DRAM, the latency penalty may be very large, since DRAM accesses are much slower than SRAM accesses.

FIG. 8 is a conceptual diagram illustrating a process for calculating device affinity in a hybrid memory device (100) according to one embodiment of the present invention.

In Figure 8, each time interval is depicted only for the purpose of explaining the concept, and each time interval is not drawn to scale.

In a typical direct-mapped DRAM cache organization, the victim is evicted when a cache miss occurs. In the process of evicting the victim in conventional technologies, the utility of the victim and the new cache line is not considered.

In an embodiment of the present invention, device sensitivity can be calculated by considering the type of the first request requested from the host (read, write, or erase), the spatial locality of the first request or the size of the first request (burst size), and the access cost of the first memory (110) and the second memory (120) (which varies depending on the type of the first request) (S620).

At this time, the physical characteristics of each of the first memory (110) and the second memory (120), the latency for row access, and the physical time required to process each request can be considered as access costs.

In a management method of a hybrid memory device (100) according to one embodiment of the present invention, the device sensitivity calculated in step (S620) can be calculated as shown in the following mathematical expression 1.

[Mathematical Formula 1]

DeviceSensitivityScore may refer to a device sensitivity score, _{Latency_SCM} may refer to latency for row access of a SCM-based first memory (110), and Latency _{_DRAM} may refer to latency for row access of a DRAM-based second memory (120).

NumColumnsAccessed is the number of columns requested at once, i.e., information corresponding to the Spatial Locality of the first request, and/or burst size.

If the Device Sensitivity score corresponding to the sequential read request of the four columns illustrated in Fig. 8 is calculated based on the mathematical expression 1, it can be calculated as the difference (=93) between the read Latency of DRAM (the "ACT" time interval of DRAM) and the read Latency of SCM (the "ACT" time interval of SCM) divided by the number of columns requested at once (=4). At this time, it can be seen that the PRECHARGE time interval for completing the memory operation and preparing the next memory operation is included in the calculation of the Latency in Fig. 8.

That is, if the first request is a request to read four consecutive columns, Device Sensitivity can be given as 93/4 = 23.

If the Device Sensitivity score corresponding to a single column write request illustrated in Fig. 8 is calculated based on the mathematical expression 1, it can be calculated as the difference (=955) between the write Latency of DRAM (the "ACT" time interval of DRAM) + Write Recovery and the write Latency of SCM (the "ACT" time interval of SCM) + Write Recovery divided by the number of columns requested at once (=1). At this time, it can be seen that the PRECHARGE time interval for completing the memory operation and preparing the next memory operation is included in the calculation of the latency in Fig. 8.

That is, if the first request is a 1-column write request, Device Sensitivity can be given as 955/1 = 955.

In an embodiment where the first memory (110) is a general non-volatile memory, the Device Sensitivity score may be even higher when erasing the first memory (110) is required.

That is, Device Sensitivity can be different depending on whether the first request is a read, a write, or a write that requires erasing. Also, since the access cost for each access is distributed depending on how many columns are sequentially accessed at a time, Device Sensitivity can be obtained by dividing the difference in Latency (+ Recovery + Precharge) according to device characteristics by the number of columns sequentially requested.

In step (S630), the average device sensitivity per channel can be compared with the device sensitivity calculated for the first request (S620) to determine whether to bypass the second memory (S680).

If the device sensitivity score is not large, the average access cost for one piece of data will not be large, so the second memory (120) can be bypassed and the first memory (110) can be directly accessed to process the first request (S680). That is, in this case, even if a cache miss occurs again, the cost of directly accessing the SCM-based first memory (110) to process the first request is relatively less than when the first request is processed via the second memory (120), so the GPU (200) may not perceive a significant performance degradation.

Referring again to FIG. 7, even in cases where it is advantageous for the first request to be processed via the second memory (120) which is a cache because the device sensitivity is high, the utility of the victim to be evicted from the second memory (120) which is a cache and the data required in the first request can be compared. In this case, it is necessary to basically consider whether the data to be evicted is hot data or cold data.

The DRAM affinity score calculated at this time can be discretized and stored in some of the fields of the ATIC structure shown in Fig. 5.

For example, in case 10-bit ECC is required for 32-byte data in Fig. 5, spare bits exist as many as the ECC bits saved by the direct-mapped technique, and the DRAM affinity level can be stored in the spare bit space. The DRAM affinity score or Device Sensitivity Score can be discretized for easy storage, comparison, and management.

In step (S640), the DRAM affinity score can be calculated by multiplying the Device Sensitivity Score of the first request by the activation counter information of the corresponding page. The activation counter of the present invention is an indicator indicating whether the corresponding page is hot data or cold data, and may be data corresponding to a type of access frequency. In one embodiment of the present invention, a per-page activation counter is managed, but in another embodiment of the present invention, the activation counter may be managed based on a size other than a page.

If the DRAM affinity level of the first request calculated in step (S640) is greater than the DRAM affinity level of the big team candidate, the first request is hotter data and is more likely to be accessed in the future, so the cache line of the second memory (120) is replaced with the data of the first request (S660), and the big team can be evicted to the first memory (110).

If the DRAM affinity level of the first request calculated in step (S640) is not greater than the DRAM affinity level of the big team candidate, the first request is colder data and is unlikely to be accessed in the future, so the big team candidate can be bypassed without being evicted (S680).

At this time, since the big team candidate remaining in the cache second memory (120) has experienced a cache miss once, the activation counter needs to be decreased. Accordingly, the DRAM affinity level of the remaining big team candidate can be decreased by a predetermined probability p _{_dec} , or an operation corresponding to the activation counter can be performed (S670).

FIG. 9 is a conceptual diagram illustrating an example of a memory controller, memory management device, or computing system within a generalized hybrid memory device capable of performing at least a portion of the processes of FIGS. 1 through 8.

The memory management device may be included within the hybrid memory device, or may be located externally to the hybrid memory device to manage/control each memory element within the hybrid memory device.

In the embodiments of FIGS. 1 to 8, although omitted in the drawings, a processor and a memory are electronically connected to each component, and the operation of each component can be controlled or managed by the processor.

At least a portion of the process of the method according to one embodiment of the present invention can be executed by the computing system (1000) of FIG. 9.

Referring to FIG. 9, a computing system (1000) according to one embodiment of the present invention may be configured to include a processor (1100), a memory (1200), a communication interface (1300), a storage device (1400), an input interface (1500), an output interface (1600), and a bus (1700).

A computing system (1000) according to one embodiment of the present invention may include at least one processor (1100) and a memory (1200) storing instructions that instruct the at least one processor (1100) to perform at least one step. At least some steps of a method according to one embodiment of the present invention may be performed by the at least one processor (1100) loading and executing instructions from the memory (1200).

The processor (1100) may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed.

Each of the memory (1200) and the storage device (1400) may be configured with at least one of a volatile storage medium and a nonvolatile storage medium. For example, the memory (1200) may be configured with at least one of a read only memory (ROM) and a random access memory (RAM).

Additionally, the computing system (1000) may include a communication interface (1300) that performs communication via a wireless network.

Additionally, the computing system (1000) may further include a storage device (1400), an input interface (1500), an output interface (1600), etc.

Additionally, each component included in the computing system (1000) may be connected to each other by a bus (1700) and communicate with each other.

Examples of the computing system (1000) of the present invention may include a desktop computer, a laptop computer, a notebook, a smart phone, a tablet PC, a mobile phone, a smart watch, a smart glass, an e-book reader, a portable multimedia player (PMP), a portable game console, a navigation device, a digital camera, a digital multimedia broadcasting (DMB) player, a digital audio recorder, a digital audio player, a digital video recorder, a digital video player, a PDA (Personal Digital Assistant), etc.

The operation of the method according to an embodiment of the present invention can be implemented as a computer-readable program or code on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices that store information that can be read by a computer system. In addition, the computer-readable recording medium can be distributed over network-connected computer systems so that the computer-readable program or code can be stored and executed in a distributed manner.

Additionally, the computer-readable recording medium may include hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. The program instructions may include not only machine language codes produced by a compiler, but also high-level language codes that can be executed by the computer using an interpreter, etc.

While some aspects of the invention have been described in the context of an apparatus, they may also represent a description of a corresponding method, wherein a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method may also be represented as a feature of a corresponding block or item or a corresponding device. Some or all of the method steps may be performed by (or using) a hardware device, such as, for example, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, at least one or more of the most important method steps may be performed by such a device.

In embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In embodiments, a field-programmable gate array may operate in conjunction with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by some hardware device.

Although the present invention has been described above with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various modifications and changes may be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below.

Claims (20)

A method for managing hybrid memory performed by a memory controller that manages hybrid memory,
A step of receiving a plurality of first data to be cached from a first memory which is a non-volatile storage class memory (SCM);
A step of collectively storing tag information of the plurality of first data in a first area within a cache line of a second memory capable of random access; and
A step of sequentially storing the plurality of first data in a second area within the cache line of the second memory;
Including,
The first memory and the second memory share a common address of a first number of bits,
The tag information of the plurality of first data includes offset information of a second number of bits between the first address excluding the common address among the first addresses on the first memory where the plurality of first data are stored and the second address on the second memory,
The tag information of the plurality of first data has a size that can be collectively stored in the first area, which is an area of a size that can be read with a single access to the second memory.
How to manage hybrid memory. In the first paragraph,
A step of storing error control information of the plurality of first data in a third area within the cache line of the second memory adjacent to the first area;
Including more,
How to manage hybrid memory. delete In the first paragraph,
A step of providing the tag information for the plurality of first data stored in the first area to the host when information on the plurality of first data is requested from the host;
Including more,
How to manage hybrid memory. In the first paragraph,
A step in which a host stores tag information for the plurality of first data stored in the first area in a part of a level 2 cache as identification information for the plurality of first data;
Including more,
How to manage hybrid memory. delete A method for managing hybrid memory performed by a memory controller that manages hybrid memory,
A step of calculating device sensitivity between the first memory and the second memory for the first request, when data corresponding to a first request of a host is not stored in a second memory capable of random access as data stored in a non-volatile first memory;
A step of determining whether to bypass the second memory and directly access the first memory for the first request based on the device sensitivity;
a step of calculating an affinity of the first request for the second memory when it is not determined to bypass the second memory based on the device sensitivity; and
A step of determining whether to replace a part of the cache line of the second memory by the first request based on a comparison result between the affinity of the first request to the second memory and the affinity of the big team data to be expelled from the second memory to the second memory;
Including,
The above device sensitivity is calculated based on a first cost of the first memory according to the type of access of the first request, and a second cost of the second memory for processing the first request.
How to manage hybrid memory. In Article 7,
In the step of calculating the above device sensitivity,
The above device sensitivity is calculated by dividing by the number of consecutive data included in the first request.
How to manage hybrid memory. In Article 7,
In the step of calculating the above device sensitivity,
The above device sensitivity is discretized into any value between a predetermined minimum value and an observed maximum value.
How to manage hybrid memory. delete In Article 7,
In the step of calculating the affinity of the above first request to the above second memory,
The affinity for the second memory is determined based on the access frequency for data corresponding to the first request and the device sensitivity of the first request.
How to manage hybrid memory. In Article 7,
In the step of calculating the affinity of the above first request to the above second memory,
The affinity for the second memory is discretized to any value between a predetermined minimum and an observed maximum.
How to manage hybrid memory. In Article 7,
A step of processing the first request by bypassing the second memory and directly accessing the first memory for the first request when it is not decided to replace a part of the cache line of the second memory by the first request;
Including more,
How to manage hybrid memory. In Article 13,
A step of reducing the affinity of the big data for the second memory according to a predetermined condition in response to an event in which data corresponding to the first request is not stored in the second memory when it is not decided to replace a part of the cache line of the second memory by the first request;
Including more,
How to manage hybrid memory. The first memory is non-volatile storage class memory (SCM);
A second memory capable of random access; and
memory controller;
Including,
The above memory controller,
Collectively storing tag information of a plurality of first data to be cached from the first memory in a first area within a cacheline of the second memory,
Sequentially storing the plurality of first data in the second area within the cache line of the second memory,
The first memory and the second memory share a common address of a first number of bits,
The tag information of the plurality of first data includes offset information of a second number of bits between the first address excluding the common address among the first addresses on the first memory where the plurality of first data are stored and the second address on the second memory,
The tag information of the plurality of first data has a size that can be collectively stored in the first area, which is an area of a size that can be read with a single access to the second memory.
Hybrid memory device. In Article 15,
The above memory controller,
Storing error control information of the plurality of first data in a third area within the cache line of the second memory adjacent to the first area,
Hybrid memory device. delete In Article 15,
The above memory controller,
When information on the plurality of first data is requested from the host, the tag information on the plurality of first data stored in the first area is provided to the host.
Hybrid memory device. In Article 18,
The above memory controller,
Providing the tag information for the plurality of first data to the host so that the host stores the tag information for the plurality of first data stored in the first area in a part of the level 2 cache as identification information for the plurality of first data.
Hybrid memory device. delete