CN116601609A - store data in computer storage - Google Patents
store data in computer storage Download PDFInfo
- Publication number
- CN116601609A CN116601609A CN202080107869.XA CN202080107869A CN116601609A CN 116601609 A CN116601609 A CN 116601609A CN 202080107869 A CN202080107869 A CN 202080107869A CN 116601609 A CN116601609 A CN 116601609A
- Authority
- CN
- China
- Prior art keywords
- data
- computer storage
- storing
- copy
- stored
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
- G06F11/1076—Parity data used in redundant arrays of independent storages, e.g. in RAID systems
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
技术领域technical field
本公开涉及将数据存储在计算机存储中。This disclosure relates to storing data in computer storage.
背景技术Background technique
存储在计算机存储系统中的数据可以通过纠删编码(erasure coding)或复制(replication)来保护数据不丢失或不损坏。纠删编码创建一种数学函数将数据的各个部分描述为数据的其它部分的函数,以使得在发生部分丢失或损坏时恢复完整数据。复制创建数据的额外副本,该额外副本可以存储在与原始数据不同的存储设备(例如,不同的磁盘驱动器)中。纠删编码用于保护数据的一个优点在于,纠删码的内存占用空间通常小于纠删码所保护的数据的占用空间。复制用于保护数据的一个优点在于,创建副本和使用该副本恢复数据的计算复杂性相对较低。Data stored in computer storage systems can be protected from loss or damage by erasure coding or replication. Erasure coding creates a mathematical function that describes parts of the data as a function of other parts of the data, so that complete data can be recovered in the event of partial loss or corruption. Replication creates an additional copy of the data, which may be stored on a different storage device (eg, a different disk drive) than the original data. One advantage of erasure coding for protecting data is that the memory footprint of an erasure code is typically smaller than the footprint of the data it protects. One advantage of replication for protecting data is that the computational complexity of creating a copy and using that copy to restore the data is relatively low.
发明内容Contents of the invention
本公开的目的是提供一种用于将数据存储在计算机存储中的方法,其中,保护所述数据不丢失或不损坏。It is an object of the present disclosure to provide a method for storing data in computer storage, wherein said data is protected from loss or corruption.
上述和其它目的通过独立权利要求的特征来实现。其它实现方式在从属权利要求、说明书和附图中是显而易见的。The above and other objects are achieved by the features of the independent claims. Other implementations are evident from the dependent claims, the description and the figures.
本公开的第一方面提供了一种用于将数据存储在计算机存储中的方法。所述方法包括:将数据存储在计算机存储中;将所述数据的纠删码存储在所述计算机存储中;将所述数据的副本存储在所述计算机存储中;以及根据所述将所述数据的副本存储在所述计算机存储中的结果,确认存储所述数据的所述纠删码的所述计算机存储可用于存储其它数据。A first aspect of the present disclosure provides a method for storing data in computer storage. The method comprises: storing data in computer storage; storing an erasure code of said data in said computer storage; storing a copy of said data in said computer storage; and storing said As a result of storing a copy of data in said computer storage, it is confirmed that said computer storage storing said erasure code for said data is available for storing other data.
在上述方法中,存储在计算机存储中的数据由数据的纠删码(例如,奇偶校验信息)保护。存储数据的纠删码可以有利地实现数据的相对快速保护。因此,数据在短期内丢失或损坏的风险可能会减少。例如,纠删码的内存占用空间相对较小,可以实现纠删码存储在本地存储中,即存储在数据本地的存储中。例如,当数据通过上述方法存储在一个或多个存储设备中时,即使一个或多个存储设备具有相对较小的备用存储容量,纠删码也可以存储在相同的一个或多个存储设备中。因此,通过避免需要将数据传输到外部备份资源以及与之相关的延迟,可以实现数据的相对快速保护。此外,由于纠删码可以存储在本地,在数据丢失时,使用纠删码恢复数据可能比从远程存储设备恢复数据相对更快。In the methods described above, data stored in computer storage is protected by an erasure code (eg, parity information) for the data. Erasure coding of stored data can advantageously enable relatively fast protection of data. Therefore, the risk of data loss or corruption in the short term may be reduced. For example, the memory footprint of the erasure code is relatively small, and the erasure code can be stored in local storage, that is, stored in the local storage of the data. For example, when data is stored in one or more storage devices by the above method, even if one or more storage devices have relatively small spare storage capacity, erasure codes can also be stored in the same one or more storage devices . Thus, relatively quick protection of data can be achieved by avoiding the need to transfer data to an external backup resource and the delays associated with it. In addition, since erasure codes can be stored locally, in the event of data loss, data recovery using erasure codes may be relatively faster than recovering data from remote storage devices.
然而,使用纠删码恢复数据在计算上可能相对复杂,因此缺点是消耗存储系统的计算资源,例如,处理器时间。因此,上述方法还包括:存储数据的副本,即复制数据。例如,数据的副本可以通过定期数据备份过程存储,由此副本数据与原始数据存储在不同的存储设备中。与纠删码相比,副本数据的创建和存储时间可能相对较长,例如,因为副本数据的内存占用空间相对较大,需要将副本数据传输到适当大的存储设备,该存储设备可以远离数据的存储位置。但是,在存储数据部分丢失或损坏时,副本数据可以有利地使得在计算上简单恢复数据。However, recovering data using erasure codes may be relatively computationally complex, and thus has the disadvantage of consuming computing resources of the storage system, eg, processor time. Therefore, the above method also includes: storing a copy of the data, that is, replicating the data. For example, a copy of data may be stored through a regular data backup process, whereby the copy data is stored on a different storage device than the original data. The creation and storage of replica data can be relatively long compared to erasure coding, for example, because the memory footprint of replica data is relatively large and the replica data needs to be transferred to a suitably large storage device that can be kept away from the data storage location. However, replicating the data can advantageously make recovering the data computationally simple in the event that stored data is partially lost or corrupted.
因此,存储数据的纠删码与随后的存储数据的副本相组合,可以有利地通过纠删编码对数据进行早期保护以及通过副本数据对数据进行增强长期保护。Thus, erasure coding of stored data combined with subsequent copies of stored data may advantageously provide early protection of data by erasure coding and enhanced long-term protection of data by replicating data.
然而,在存储数据的副本之后,即在复制数据之后,可以认为纠删码是多余的,因为随后在数据丢失或损坏的情况下,可以使用数据的副本来恢复数据。同时,纠删码占用存储空间,这在纠删码存储在存储容量相对较小的存储设备中时,会特别成问题。因此,上述方法包括又一步骤:根据将数据的副本存储在计算机存储中的结果,确认存储纠删码的计算机存储可用于存储其它数据。换句话说,在上述方法中,一旦复制完数据并存储好副本数据,上述方法包括:确认纠删码占用的存储空间可由新数据覆盖。在示例中,上述方法甚至还可以包括:在存储副本数据之后,从存储中擦除纠删码。因此,通过上述方法的这一步骤,数据和保护配置的总内存占用空间可以有利地减少。但是,由于上述方法的这一步骤是根据存储数据的副本的结果执行的,因此上述方法是安全的,因为该步骤的执行取决于副本数据的存储。因此,数据既不存在纠删码也不存在副本数据的风险得到减少。However, after a copy of the data is stored, that is, after the data has been replicated, erasure coding can be considered redundant, since the copy of the data can then be used to recover the data in the event of data loss or corruption. At the same time, erasure codes take up storage space, which can be particularly problematic when erasure codes are stored in storage devices with relatively small storage capacities. Accordingly, the method described above includes the further step of confirming that the computer storage storing the erasure code is available for storing other data as a result of storing a copy of the data in the computer storage. In other words, in the above method, once the data is copied and the copy data is stored, the above method includes: confirming that the storage space occupied by the erasure code can be covered by new data. In an example, the above method may even include: erasing the erasure code from the storage after storing the copy data. Thus, by this step of the method described above, the total memory footprint of data and protection configurations can advantageously be reduced. However, since this step of the above method is performed as a result of storing a copy of the data, the above method is safe because the execution of this step depends on the storage of the copy data. Therefore, the risk of data having neither erasure codes nor duplicate data is reduced.
在示例中,所述方法可以包括:通过公共存储设备存储数据、纠删码和副本数据中的两个或两个以上,例如,在同一磁盘驱动器中的不同位置上。在其它示例中,数据、纠删码和副本数据中的两个或两个以上可以由相互不同的存储设备存储。例如,数据和纠删码可以存储在一个或多个存储设备(例如,磁盘驱动器)组成的第一组中,而副本数据可以存储在一个或多个不同的存储设备(例如,磁盘驱动器)中。In an example, the method may include storing two or more of the data, the erasure code, and the duplicate data via a common storage device, eg, at different locations on the same disk drive. In other examples, two or more of the data, the erasure code, and the duplicate data may be stored by mutually different storage devices. For example, data and erasure codes may be stored on a first set of one or more storage devices (e.g., disk drives), while duplicate data may be stored on one or more different storage devices (e.g., disk drives) .
上述方法可以包括以下步骤:在存储纠删码之前,创建该纠删码,例如,奇偶校验信息。用于为数据创建纠删码的方法是本领域技术人员已知的。The above method may include the step of creating an erasure code, eg, parity information, before storing the erasure code. Methods for creating erasure codes for data are known to those skilled in the art.
在本说明书的上下文中,除非另有说明,术语“存储器”(例如,计算机存储器)和“内存”(例如,计算机内存)都用于泛指用于存储数据的存储资源。在这方面,每个术语都旨在包括用于长期存储数据的存储资源(例如,磁盘驱动器、固态驱动器或闪存)以及用于短期存储数据的存储资源(例如,随机存取存储器)。In the context of this specification, unless otherwise stated, the terms "storage" (eg, computer memory) and "internal memory" (eg, computer memory) are used to refer generally to storage resources used to store data. In this regard, each term is intended to include storage resources used for long-term storage of data (eg, magnetic disk drives, solid-state drives, or flash memory) as well as storage resources used for short-term storage of data (eg, random access memory).
在一种实现方式中,所述将数据存储在计算机存储中包括:将所述数据的第一段存储在第一计算机存储设备中,以及将所述数据的第二段存储在第二计算机存储设备中。In one implementation, said storing data in computer storage includes: storing a first segment of said data in a first computer storage device, and storing a second segment of said data in a second computer storage device. in the device.
换句话说,上述方法可以包括:将数据分布在多个存储设备中,以下称为分带。数据的分带可以有利地提高数据读取/写入性能,这是因为数据的各段可以同时从它们各自的存储设备中读取/写入它们各自的存储设备中。此外,在多个设备上对数据进行分带减少了在存储设备的子集发生故障时会丢失的数据的比例。因此,纠删码只对丢失相对较少的数据具有适应能力,这通常是可接受的。这可以有利地降低纠删码的复杂性,从而降低生成纠删码和使用纠删码来恢复数据的复杂性。在示例中,上述方法可以包括:将数据存储在两个以上段中,可选地存储在两个以上存储设备中。例如,在示例中,上述方法可以包括:将数据以五个或六个段存储在五个或六个存储设备中。上述方法可以包括以下步骤:在存储数据之前,将该数据分割成两个或两个以上段。In other words, the above method may include: distributing data among multiple storage devices, which is called striping hereinafter. Striping of data can advantageously improve data read/write performance because segments of data can be simultaneously read from/written to their respective storage devices. Additionally, striping data across multiple devices reduces the proportion of data that would be lost if a subset of storage devices failed. Therefore, erasure codes are only resilient to loss of relatively small amounts of data, which is usually acceptable. This can advantageously reduce the complexity of erasure codes, thereby reducing the complexity of generating and using erasure codes to recover data. In an example, the above method may include storing data in two or more segments, optionally in two or more storage devices. For example, in an example, the above method may include: storing data in five or six segments in five or six storage devices. The above method may include the following steps: before storing the data, divide the data into two or more segments.
在一种实现方式中,所述将所述数据的副本存储在所述计算机存储中包括:将所述数据的副本存储在与存储所述数据的计算机存储设备分开和/或与存储所述纠删码的计算机存储设备分开的一个或多个计算机存储设备中。In one implementation, the storing a copy of the data in the computer storage includes: storing a copy of the data in a separate computer storage device from which the data is stored and/or from a storage device storing the corrected data. Erasure coded computer storage devices separate one or more computer storage devices.
换句话说,在示例中,上述方法包括:将副本数据存储在物理上与存储数据的存储设备和存储纠删码的存储设备中的至少一个分开的存储设备中。副本数据与数据和/或纠删码的这种分开有利地降低了在存储设备发生故障时丢失副本数据和数据或纠删码的可能性。在示例中,数据和纠删码可以存储在一个或多个存储设备(例如,硬盘)中,而副本数据可以存储在一个或多个单独的存储设备中。In other words, in an example, the above method includes: storing the duplicate data in a storage device physically separate from at least one of the storage device storing the data and the storage device storing the erasure code. This separation of replica data from data and/or erasure codes advantageously reduces the likelihood of losing replica data and data or erasure codes in the event of a storage device failure. In an example, data and erasure codes may be stored on one or more storage devices (eg, hard drives), while duplicate data may be stored on one or more separate storage devices.
在一种实现方式中,所述方法还包括:将标识所述数据的所述副本的位置的位置数据存储在所述计算机存储中。In one implementation, the method further includes storing location data identifying a location of the copy of the data in the computer storage.
换句话说,位置数据限定副本数据在计算机存储中的机器可读地址。例如,位置数据可以标识副本数据所在的存储设备编号和/或偏移。因此,位置数据可以有利地使得副本数据从计算机存储中容易检索到,这在数据丢失时可能是有用的。例如,位置数据可以由内存控制器在存储副本数据时生成。In other words, the location data defines the machine-readable address of the replica data in computer storage. For example, the location data may identify the storage device number and/or offset where the duplicate data resides. Thus, location data may advantageously enable easy retrieval of duplicate data from computer storage, which may be useful in the event of data loss. For example, location data can be generated by the memory controller when storing replica data.
在一种实现方式中,所述将标识所述数据的所述副本的位置的位置数据存储在所述计算机存储中包括:将所述位置数据存储在存储所述数据的所述副本的计算机存储设备中。In one implementation, said storing location data identifying a location of said copy of said data in said computer storage comprises: storing said location data in said computer storage storing said copy of said data in the device.
换句话说,位置数据可以与副本数据的至少一部分存储在相同的存储设备中。这种设置的优点在于,位置数据可以易于由用于访问副本数据的内存控制器访问。因此,在数据丢失导致需要副本数据时,同一内存控制器可以读取位置数据并检索副本数据。这可以有利地减少使用副本数据来恢复数据所需的时间。例如,副本数据可以存储在单个存储设备中,而位置数据可以存储在该相同的存储设备中。在这一示例中,位置数据可以标识副本数据在存储设备中的偏移。在另一个示例中,副本数据可以分布在多个存储设备上,而位置数据可以存储在这些多个设备中的一个设备中。In other words, the location data may be stored on the same storage device as at least a portion of the replica data. The advantage of this setup is that the location data can be easily accessed by the memory controller used to access the replica data. Therefore, the same memory controller can read the location data and retrieve the duplicate data when a data loss causes the duplicate data to be needed. This can advantageously reduce the time required to restore data using duplicate data. For example, duplicate data may be stored on a single storage device, while location data may be stored on that same storage device. In this example, the location data may identify an offset within the storage device of the duplicate data. In another example, replica data may be distributed across multiple storage devices, and location data may be stored on one of the multiple devices.
在一种实现方式中,所述方法还包括:根据所述将所述数据的副本存储在所述计算机存储中的结果,将标记存储所述数据的所述计算机存储的标志存储在所述计算机存储中。In one implementation, the method further includes storing in the computer storage a flag marking the computer storage storing the data as a result of the storing a copy of the data in the computer storage in storage.
换句话说,上述方法可以包括:在存储副本数据之后,标记存储中存储数据的位置。因此,标志可以标识计算机存储的位置,包括全部或部分数据,这些数据已经通过复制(即通过存储副本数据)得到保护。因此,标志可以用于方便地确认是否已经复制数据,即是否已经存储副本数据。这样做的一个优点在于,内存管理器能够简单地通过读取标志来确定是否已经复制数据,因此,在数据丢失时可以知道是否可以检索到副本数据。In other words, the above method may include: after storing the replica data, marking a location in the storage where the data is stored. Thus, a flag can identify a location of computer storage, including all or part of data, which has been protected by replication (i.e. by storing duplicate data). Therefore, the flag can be used to easily confirm whether the data has been copied, that is, whether the duplicate data has been stored. One advantage of this is that the memory manager can simply read the flag to determine whether the data has been copied, so in the event of data loss it knows whether the copied data can be retrieved.
在一种实现方式中,所述方法还包括:将其它数据存储在确认可用于存储其它数据的所述计算机存储中。换句话说,上述方法还可以包括:在复制数据之后,覆盖所述计算机存储中存储所述纠删码的一个或多个位置。In one implementation, the method further includes storing other data in the computer storage identified as being available for storing other data. In other words, the above method may further include: after copying the data, overwriting one or more locations in the computer storage where the erasure code is stored.
在一种实现方式中,所述将所述数据的纠删码存储在所述计算机存储中包括:将所述纠删码存储在存储所述数据的一个或多个计算机存储设备中。换句话说,纠删码可以与数据存储在相同的一个或多个存储设备中。鉴于纠删码是根据数据生成的,这种设置可以减少生成和存储纠删码所需的时间,因为纠删码生成器可以从相同的一个或多个存储设备中读取和写入相同的一个或多个存储设备中。In one implementation, the storing the erasure code of the data in the computer storage includes: storing the erasure code in one or more computer storage devices storing the data. In other words, erasure codes can be stored on the same storage device or devices as the data. Given that erasure codes are generated from data, this setup reduces the time required to generate and store erasure codes, since the erasure code generator can read from and write to the same storage device or devices. one or more storage devices.
在一种实现方式中,所述将所述数据的纠删码存储在所述计算机存储中包括:将第一纠删码和第二纠删码存储在所述计算机存储中,其中,所述第一纠删码和所述第二纠删码相互独立;根据所述在所述将所述数据的副本存储在所述计算机存储中的结果,确认存储所述数据的所述纠删码的所述计算机存储可用于存储其它数据包括:根据所述将所述数据的副本存储在所述计算机存储中的结果,确认只有存储所述数据的所述第二纠删码的所述计算机存储可用于存储其它数据。In an implementation manner, the storing the erasure code of the data in the computer storage includes: storing the first erasure code and the second erasure code in the computer storage, wherein the The first erasure code and the second erasure code are independent of each other; and according to the result of said storing a copy of said data in said computer storage, validating said erasure code storing said data Said computer storage being available to store other data comprises: as a result of said storing a copy of said data in said computer storage, confirming that only said computer storage storing said second erasure code of said data is available to store other data.
换句话说,上述方法可以包括:存储两个相互独立的纠删码,即纠删码都可在数据的各个部分上独立于其它纠删码进行操作,以重新创建数据。与单个纠删码相比,为数据创建/存储两个纠删码可以有利地提高数据适应能力,因为其中一个纠删码本身可能会丢失或损坏,而数据仍然可以使用剩余的纠删码来恢复。然而,在上述方法的所述实现方式中,在复制数据之后只选择覆盖其中一个纠删码(即第二纠删码),而将第二纠删码保留在存储中。这种设置可以有利地平衡数据恢复能力与保护的内存占用空间。在生成多个纠删码的其它示例中,可以在复制数据之后选择覆盖所有纠删码。In other words, the above method may include: storing two mutually independent erasure codes, that is, each erasure code can operate independently of other erasure codes on each part of the data, so as to recreate the data. Creating/storing two erasure codes for data can advantageously improve data resilience compared to a single erasure code, since one of the erasure codes itself may be lost or corrupted, while the data can still be recovered using the remaining erasure code recover. However, in the implementation of the above method, only one of the erasure codes (that is, the second erasure code) is selected to be overwritten after the data is copied, and the second erasure code is kept in storage. This setup provides a favorable balance of data resiliency versus protected memory footprint. In other examples where multiple erasure codes are generated, one may choose to overwrite all erasure codes after copying the data.
在一种实现方式中,所述方法还包括:对存储在所述计算机存储中的所述数据执行错误检测操作;响应于检测到存储在所述计算机存储中的所述数据存在错误,检索存储在所述计算机存储中的所述数据的所述副本。In one implementation, the method further includes: performing an error detection operation on the data stored in the computer storage; in response to detecting an error in the data stored in the computer storage, retrieving the stored said copy of said data in said computer storage.
换句话说,在示例中,上述方法可以包括:测试数据以检查错误,例如,数据的部分或全部丢失或损坏,并且响应于检测到数据丢失/损坏,上述方法可以包括:检索副本数据。这种“主动”测试数据和检索副本数据的过程可以减少数据丢失和使用副本数据来恢复数据之间的时间,从而可以有利地最小化客户端在数据丢失后访问数据的时间。In other words, in an example, the above method may include testing the data to check for errors, eg, partial or total loss or corruption of the data, and in response to detecting data loss/corruption, the above method may include retrieving duplicate data. This process of "proactively" testing data and retrieving replica data can reduce the time between data loss and using the replica data to restore the data, thereby advantageously minimizing the time for clients to access the data after data loss.
本公开的第二方面提供了一种计算机程序。所述计算机程序包括指令,当所述指令由处理器执行时,使得所述处理器执行以下操作:将数据存储在计算机存储中;将所述数据的纠删码存储在所述计算机存储中;将所述数据的副本存储在所述计算机存储中;根据所述将所述数据的副本存储在所述计算机存储中的结果,确认存储所述数据的所述纠删码的所述计算机存储可用于存储其它数据。A second aspect of the disclosure provides a computer program. The computer program includes instructions that, when executed by a processor, cause the processor to: store data in computer storage; store an erasure code for the data in the computer storage; storing a copy of said data in said computer storage; confirming that said computer storage storing said erasure code of said data is available as a result of said storing said copy of said data in said computer storage to store other data.
在所述第二方面的一种实现方式中,所述计算机程序包括指令,当所述指令由处理器执行时,使得所述处理器执行本公开第一方面的任一种实现方式提供的方法。In an implementation manner of the second aspect, the computer program includes instructions, and when the instructions are executed by a processor, the processor executes the method provided by any implementation manner of the first aspect of the present disclosure. .
本公开的第三方面提供了一种计算机可读数据载体。所述计算机可读数据载体中存储有本公开第二方面提供的计算机程序。A third aspect of the disclosure provides a computer readable data carrier. The computer program provided by the second aspect of the present disclosure is stored in the computer-readable data carrier.
本公开的第四方面提供了一种计算系统。所述计算系统包括计算机存储和耦合到所述计算机存储的处理器,所述处理器用于:将数据存储在计算机存储中;将所述数据的纠删码存储在所述计算机存储中;将所述数据的副本存储在所述计算机存储中;根据所述将所述数据的副本存储在所述计算机存储中的结果,确认存储所述数据的所述纠删码的所述计算机存储可用于存储其它数据。A fourth aspect of the present disclosure provides a computing system. The computing system includes computer storage and a processor coupled to the computer storage, the processor configured to: store data in the computer storage; store an erasure code for the data in the computer storage; a copy of said data is stored in said computer storage; and based on the result of said storing a copy of said data in said computer storage, confirming that said computer storage storing said erasure code for said data is available for storage other data.
在所述第三方面的一种实现方式中,所述处理器用于执行本公开第一方面的任一种实现方式提供的方法。In an implementation manner of the third aspect, the processor is configured to execute the method provided in any implementation manner of the first aspect of the present disclosure.
本公开的这些和其它方面在下面描述的一个或多个实施例中是显而易见的。These and other aspects of the disclosure are apparent in one or more of the embodiments described below.
附图说明Description of drawings
为了更容易理解本公开,下面结合附图通过举例描述本公开实施例,在附图中:In order to understand the present disclosure more easily, embodiments of the present disclosure are described below by way of examples in conjunction with the accompanying drawings, in which:
图1是体现本公开一方面的计算系统的一个示例的示意图;FIG. 1 is a schematic diagram of one example of a computing system embodying an aspect of the present disclosure;
图2是计算系统中的计算机内存的示意图;2 is a schematic diagram of computer memory in a computing system;
图3是计算系统中的计算机存储装置的示意图;3 is a schematic diagram of a computer storage device in a computing system;
图4是用于将数据存储在参考图3标识的计算机存储装置中的示例性方法中包括的过程的示意图,所述方法包括分割数据进行存储的方法;4 is a schematic diagram of processes involved in an exemplary method for storing data in a computer storage device identified with reference to FIG. 3, the method including a method of partitioning data for storage;
图5是分割数据进行存储的方法中包括的过程的示意图;FIG. 5 is a schematic diagram of a process involved in a method of segmenting data for storage;
图6是存储位置数据的过程的示意图;6 is a schematic diagram of a process of storing location data;
图7是将标志存储在计算机存储中的过程的示意图;Figure 7 is a schematic diagram of the process of storing the flags in computer storage;
图8是确认存储纠删码的计算机存储可用于存储其它数据的过程的示意图;Figure 8 is a schematic diagram of a process for confirming that computer storage storing erasure codes can be used to store other data;
图9是将其它数据存储在计算机存储的已标识位置上的过程的示意图;以及Figure 9 is a schematic illustration of the process of storing other data at identified locations in computer storage; and
图10是执行错误检测操作的过程的示意图。FIG. 10 is a schematic diagram of a process of performing an error detection operation.
具体实施方式Detailed ways
首先统一参考图1和图2,体现本公开一方面的一个示例的计算系统101包括多个客户端设备102和103、计算机存储系统104和备份存储系统105。客户端设备102和103通过网络106以通信方式耦合到计算机存储系统104。计算机存储系统104通过网络107以通信方式耦合到备份存储系统105。Referring initially to FIGS. 1 and 2 collectively, an example computing system 101 embodying an aspect of the present disclosure includes a plurality of client devices 102 and 103 , a computer storage system 104 and a backup storage system 105 . Client devices 102 and 103 are communicatively coupled to computer storage system 104 through network 106 . Computer storage system 104 is communicatively coupled to backup storage system 105 through network 107 .
客户端设备102和103使用计算机存储系统104来存储数据。例如,客户端设备102和103可以通过网络106向计算机存储系统104发送存储或检索数据的请求,并且可以类似地通过网络106与计算机存储系统104交换用于存储或检索的数据。客户端设备102和103可以是,例如,台式计算机、便携式计算机或智能手机。客户端设备102和103可以包括计算功能,例如,计算机处理器。在上述示例中,两个客户端设备102和103是描述为耦合到计算机存储系统104。在其它示例中,耦合到计算机存储系统104的客户端设备的数量可以大于或小于两个。客户端设备102和103可以远离计算机存储系统104,实际上是彼此相对远离。例如,计算机存储系统104可以位于中央数据中心内。因此,客户端设备102和103可以使用计算机存储系统104中的存储资源来存储数据。Client devices 102 and 103 use computer storage system 104 to store data. For example, client devices 102 and 103 may send requests to computer storage system 104 over network 106 to store or retrieve data, and may similarly exchange data with computer storage system 104 over network 106 for storage or retrieval. Client devices 102 and 103 may be, for example, desktop computers, laptop computers, or smartphones. Client devices 102 and 103 may include computing functionality, such as a computer processor. In the above example, two client devices 102 and 103 are depicted as being coupled to computer storage system 104 . In other examples, the number of client devices coupled to computer storage system 104 may be greater or less than two. Client devices 102 and 103 may be remote from computer storage system 104, and indeed relatively remote from each other. For example, computer storage system 104 may be located in a central data center. Accordingly, client devices 102 and 103 may use storage resources in computer storage system 104 to store data.
计算机存储系统104用于为多个客户端设备(例如,客户端设备102和103)提供存储资源。如下所述,在上述示例中,计算机存储系统104用于与客户端设备102和103进行通信以存储和检索数据,还用于与备份存储系统105进行通信以存储在计算机存储系统104中的数据的副本,即存储到备份存储系统104中。Computer storage system 104 is used to provide storage resources for multiple client devices (eg, client devices 102 and 103). In the above example, computer storage system 104 is used to communicate with client devices 102 and 103 for storing and retrieving data, and also to communicate with backup storage system 105 for storing data in computer storage system 104, as described below. A copy of , that is, stored in the backup storage system 104.
计算机存储系统104包括处理器108、内存109、存储器110、输入/输出接口111和系统总线112。处理器108用于控制计算机存储系统的操作,例如,处理客户端设备102和103的存储和检索请求。如本文所述,在示例中,处理器108用于根据存储在内存109中的数据存储计算机程序,控制数据存储操作的过程。内存109被配置为非易失性读取/写入存储器,用于存储由处理器108执行的计算机程序和与由处理器108执行的操作相关联的操作数据。在示例中,内存109存储有用于控制数据存储的计算机程序201。在示例中,内存109是闪存,但是在其它示例中,闪存可以由替代形式的内存替代。存储器110用于存储数据,例如,用于存储客户端设备102和103的数据。在示例中,如下参考后面的附图详述,存储器110包括多个存储设备,在示例中,这些存储设备是多个磁盘驱动器。输入/输出接口111用于将客户端设备102和103连接到计算机存储系统104,以及用于将计算机存储系统104连接到备份存储和重复数据删除系统105。计算机104中的组件108至111通过系统总线112进行通信。Computer storage system 104 includes processor 108 , memory 109 , storage 110 , input/output interface 111 , and system bus 112 . Processor 108 is used to control the operation of the computer storage system, eg, to process storage and retrieval requests from client devices 102 and 103 . As described herein, in an example, processor 108 is configured to control the course of data storage operations according to a data storage computer program stored in memory 109 . The memory 109 is configured as a nonvolatile read/write memory for storing computer programs executed by the processor 108 and operation data associated with operations performed by the processor 108 . In an example, the memory 109 stores a computer program 201 for controlling data storage. In an example, memory 109 is flash memory, but in other examples, flash memory may be replaced by an alternative form of memory. The memory 110 is used for storing data, for example, for storing data of the client devices 102 and 103 . In an example, as described in detail below with reference to subsequent figures, memory 110 includes a plurality of storage devices, which in an example are a plurality of disk drives. Input/output interface 111 is used to connect client devices 102 and 103 to computer storage system 104 , and to connect computer storage system 104 to backup storage and deduplication system 105 . Components 108 through 111 in computer 104 communicate over system bus 112 .
备份存储系统105用于向计算机存储系统104提供备份存储资源,即用于存储计算机存储系统104中存储的数据的一个或多个副本。数据的备份存储提供了有用的冗余,以使得在原始数据丢失或损坏时(例如,在存储器110发生故障时)恢复数据。备份存储系统105包括处理器112、存储器113、输入/输出接口115和系统总线116。处理器112用于控制备份存储系统105的操作。如本文所述,在示例中,处理器112用于控制与计算机存储系统104进行通信,以及用于与计算机存储系统104合作控制数据备份操作的过程。存储器113用于数据的非易失性存储,例如,用于存储从计算机存储系统104接收到的数据的副本以进行备份。在示例中,存储器113包括一个或多个磁盘驱动器。输入/输出接口115用于将备份存储系统105连接到计算机存储系统104。组件112至115通过系统总线116进行通信。The backup storage system 105 is used to provide backup storage resources to the computer storage system 104 , that is, to store one or more copies of data stored in the computer storage system 104 . Backup storage of data provides useful redundancy, allowing data to be recovered if the original data is lost or corrupted (eg, in the event of memory 110 failure). The backup storage system 105 includes a processor 112 , a memory 113 , an input/output interface 115 and a system bus 116 . The processor 112 is used to control the operation of the backup storage system 105 . As described herein, in an example, processor 112 is used to control the process of communicating with computer storage system 104 and to cooperate with computer storage system 104 to control data backup operations. Memory 113 is used for non-volatile storage of data, eg, for storing copies of data received from computer storage system 104 for backup. In an example, storage 113 includes one or more disk drives. Input/output interface 115 is used to connect backup storage system 105 to computer storage system 104 . Components 112 to 115 communicate via system bus 116 .
备份存储系统105可以远离计算机存储系统104。实际上,为了提供计算机存储系统104的备份存储,可能希望备份存储系统105包括物理存储资源,例如,一个或多个磁盘驱动器,该物理存储资源与计算机存储系统104中的存储资源分开,以避免两个存储资源同时发生故障的风险。具体地,在一些示例中,备份存储系统105在地理上远离存储系统104可能是有利的,从而减少两个存储系统同时发生故障的风险。Backup storage system 105 may be remote from computer storage system 104 . In practice, to provide backup storage for computer storage system 104, it may be desirable for backup storage system 105 to include physical storage resources, such as one or more disk drives, separate from the storage resources in computer storage system 104 to avoid Risk of simultaneous failure of two storage resources. Specifically, in some examples it may be advantageous for backup storage system 105 to be geographically remote from storage system 104, thereby reducing the risk of simultaneous failure of both storage systems.
在示例中,网络106和107都可以由互联网等广域网(wide area network,WAN)、局域网(local area network,LAN)、城域网(metropolitan area network,MAN)和/或个人局域网(personal area network,PAN)等来实现。上述两个网络可以使用有线技术(例如,以太网、有线数据传输业务接口规范(Data Over Cable Service InterfaceSpecification,DOCSIS)、同步光纤网(synchronous optical networking,SONET)和/或同步数字体系(synchronous digital hierarchy,SOH),等等)和/或无线技术(例如,电气和电子工程师学会(Institute of Electrical and Electronics,IEEE)802.11(Wi-Fi)、IEEE 802.15(WiMAX)、蓝牙、ZigBee、近场通信(near-field communication,NFC)和/或长期演进(Long-Term Evolution,LTE),等等)来实现。上述两个网络可以包括用于在网络中进行数据通信的至少一个设备。例如,网络106和107都可以包括计算设备、路由器、交换机、网关、接入点和/或调制解调器。In an example, both the networks 106 and 107 can be composed of a wide area network (wide area network, WAN) such as the Internet, a local area network (local area network, LAN), a metropolitan area network (metropolitan area network, MAN) and/or a personal area network (personal area network). , PAN) etc. to achieve. The above two networks may use wired technologies (e.g., Ethernet, Data Over Cable Service Interface Specification (DOCSIS), Synchronous Optical Networking (SONET) and/or Synchronous Digital Hierarchy , SOH), etc.) and/or wireless technologies (e.g., Institute of Electrical and Electronics (IEEE) 802.11 (Wi-Fi), IEEE 802.15 (WiMAX), Bluetooth, ZigBee, Near Field Communication ( near-field communication, NFC) and/or long-term evolution (Long-Term Evolution, LTE), etc.). The above two networks may include at least one device for data communication in the network. For example, both networks 106 and 107 may include computing devices, routers, switches, gateways, access points, and/or modems.
接下来参考图3,在示例中,计算机存储系统104中的存储器110包括用于存储数据的多个存储设备。在上述示例中,存储器110包括五个分开的存储设备301至305,在示例中,这些存储设备都是磁盘驱动器。五个存储设备301至305通过系统总线306耦合到系统总线116,从而用作可由处理器108单独寻址的逻辑单元。在示例中,如下参考后面的附图详述,由处理器108运行的计算机程序201使得数据以多个段存储在存储器110中,数据结构中的每个段都存储在多个磁盘驱动器中的一个不同的磁盘驱动器中。在示例中,如下所述,例如,通过多个存储设备上的数据段的块级分带,将数据段存储在独立磁盘冗余阵列(redundant array of independent disk,RAID)级别6配置中,其中,两个校验块分布在存储设备上。Referring next to FIG. 3 , in an example, memory 110 in computer storage system 104 includes a plurality of storage devices for storing data. In the above example, memory 110 includes five separate storage devices 301 to 305, which in the example are disk drives. Five memory devices 301 through 305 are coupled to system bus 116 through system bus 306 to act as logical units that are individually addressable by processor 108 . In an example, as described in detail below with reference to subsequent figures, computer program 201 executed by processor 108 causes data to be stored in memory 110 in multiple segments, with each segment in the data structure being stored on multiple disk drives. on a different disk drive. In an example, the data segments are stored in a redundant array of independent disks (RAID) level 6 configuration, e.g., by block-level striping of the data segments across multiple storage devices, as described below, where , two check blocks are distributed on the storage device.
因此,在上述示例中,数据项(例如,文件)A到E都存储在存储器110中。数据项A到E都分割成三个段或块1至3,这些段或块的大小可以相等。对于每个数据项A至E,组成段或块1至3在三个存储设备相应组上分带,即分布,使得每个存储设备最多保存数据项A至E各自的三个段/块中的一个。另外,在示例中,对于每个数据项A至E,两个纠删码(例如,奇偶校验Pp和Pq)存储在存储器110中。在示例中,对于每个数据项A至E,数据段/块1至3和各自的奇偶校验Pp和Pq分布在存储设备301至305上,使得每个存储设备最多保存这些数据段/块或校验中的一个。Thus, in the above example, data items (eg, files) A to E are all stored in memory 110 . Data items A to E are each divided into three segments or blocks 1 to 3, which may be of equal size. For each data item A to E, the constituent segments or blocks 1 to 3 are striped, i.e. distributed, over respective groups of three storage devices such that each storage device holds at most three segments/blocks of data items A to E respectively one of. In addition, in an example, for each data item A to E, two erasure codes (eg, parities Pp and Pq) are stored in the memory 110 . In the example, for each data item A to E, data segments/blocks 1 to 3 and respective parities Pp and Pq are distributed across storage devices 301 to 305 such that each storage device holds at most these data segments/blocks or one of checksums.
如下详述,每个数据项A至E的校验Pp和Pq被配置为相互独立,使得奇偶校验Pp和Pq都可以在各自的数据段/块1至3的子集上独立于其它校验进行操作,从而恢复数据。这种配置的优点在于,存储器110对丢失段/块的子集或相应校验具有相对较强的恢复能力。具体地,在上述配置中,对于每个数据项A至E,存储器对段/块1至3中的至多两个或奇偶校验Pp和Pq的丢失具有恢复能力,这是因为数据通常可以从块/段或奇偶校验中的任意三个来重建。As detailed below, the parities Pp and Pq of each data item A to E are configured independently of each other such that both parities Pp and Pq can be independent of other parities on their respective subsets of data segments/blocks 1 to 3. Perform operations to restore data. An advantage of this configuration is that the memory 110 has a relatively strong recovery capability for a subset of lost segments/blocks or corresponding parities. Specifically, in the configuration described above, for each data item A to E, the memory is resilient to the loss of at most two of the segments/blocks 1 to 3 or parities Pp and Pq, since data can usually be retrieved from Any three of block/segment or parity to rebuild.
接下来参考图4,在示例中,存储在计算机存储系统104的内存109中的计算机程序201由计算机存储系统104执行,使得处理器108执行包括四个阶段的方法。Referring next to FIG. 4 , in an example, computer program 201 stored in memory 109 of computer storage system 104 is executed by computer storage system 104 to cause processor 108 to perform a method comprising four stages.
在阶段401中,计算机程序201可以使得处理器108将数据存储在存储系统103的存储器110中。例如,阶段401可以响应于存储系统从客户端设备102和103中的一个客户端设备接收到的存储请求(即相应客户端设备要将数据存储在存储系统104中的请求)而启动。响应于这一请求,处理器108则可以将接收到的数据存储在存储器110中。在示例中,如上参考图3所述,并且如图5详细示出,阶段401可以包括:处理器108将接收到的数据的各项(例如,文件)分割成多个段,以在存储器110的存储设备301至305上对这些段进行分带。因此,例如,阶段401可以包括:处理器108将接收到的数据的多个文件(例如,文件A至E)都分割成多个块/段,例如,三个块/段1至3,并且在存储设备301至305中的三个存储设备上对这些段进行分带。因此,在示例中,阶段401可以包括三个阶段501至503,以对数据进行分带,如图5所示。In stage 401 , the computer program 201 may cause the processor 108 to store data in the memory 110 of the storage system 103 . For example, stage 401 may be initiated in response to a storage request received by the storage system from one of client devices 102 and 103 , ie a request by the corresponding client device to store data in storage system 104 . In response to this request, processor 108 may then store the received data in memory 110 . In an example, as described above with reference to FIG. 3, and as shown in detail in FIG. These segments are striped on the storage devices 301 to 305 of . Thus, for example, stage 401 may comprise that the processor 108 divides each of the plurality of files of received data (e.g., files A to E) into a plurality of blocks/segments, e.g., three blocks/segments 1 to 3, and The segments are striped across three of the storage devices 301-305. Therefore, in an example, stage 401 may include three stages 501 to 503 to strip data, as shown in FIG. 5 .
在阶段402中,计算机程序201可以使得处理器108为在阶段401中存储的数据创建纠删码,例如,奇偶校验,并且将纠删码也存储在存储器110中。在示例中,阶段402可以包括:处理器108为数据创建两个相互独立的纠删码。用于为数据创建纠删码(例如,奇偶校验)的合适过程是本领域技术人员已知的。例如,对于每个数据项(例如,文件)A至E,奇偶校验“p”和“q”可以使用两个不同的函数来计算,例如,奇偶校验“p”可以使用XOR函数根据数组数据来计算,校验“q”可以使用里德所罗门(Reed-Solomon)码来计算。因此,纠删码(例如,奇偶校验)最多可以对存储设备301至305中的两个存储设备发生故障具有恢复能力。In stage 402 , computer program 201 may cause processor 108 to create an erasure code, eg a parity check, for the data stored in stage 401 and also store the erasure code in memory 110 . In an example, stage 402 may include the processor 108 creating two mutually independent erasure codes for the data. Suitable procedures for creating erasure codes (eg, parity) for data are known to those skilled in the art. For example, for each data item (e.g., file) A to E, the parity "p" and "q" can be calculated using two different functions, for example, the parity "p" can be calculated using the XOR function according to the array data, the check "q" can be calculated using a Reed-Solomon (Reed-Solomon) code. Therefore, erasure codes (eg, parity checks) can be resilient to failures of at most two of the storage devices 301-305.
在阶段403中,计算机程序201可以通过在备份存储系统105的存储器113中创建和存储在阶段401中存储的数据的副本,使得处理器108执行复制过程。在示例中,执行阶段402和阶段403之间可能存在延迟。在示例中,例如,为了响应客户端设备102、103的重复存储请求,上述方法中的阶段401和阶段402可以重复执行,而阶段403只能在执行阶段401和阶段402的一些情况之后执行。例如,阶段403的备份操作可以周期性地执行,例如,每天执行,使得在该期间内存储在存储系统104中的所有数据的副本可以在单个时间步长内备份。这可能是一种计算高效的数据复制方法。因此,在阶段403中,存储系统104中的处理器108可以在一个或多个阶段401中创建存储在存储器110中的一个或多个数据项的副本,并且可以通过网络107将副本数据发送给备份存储系统105。存储系统104中的处理器108可以与备份存储系统105中的处理器112进行通信,以使得备份存储系统105将副本数据保存在存储器113中。阶段403可以包括:备份存储系统105将报告发回存储系统104,以确认副本数据的存储。In stage 403 , computer program 201 may cause processor 108 to perform a copying process by creating and storing in memory 113 of backup storage system 105 a copy of the data stored in stage 401 . In an example, there may be a delay between performing stage 402 and stage 403 . In an example, stage 401 and stage 402 in the above method may be executed repeatedly, for example, in response to repeated storage requests from client devices 102, 103, while stage 403 may only be executed after some instances of stage 401 and stage 402 are executed. For example, the backup operation of stage 403 may be performed periodically, eg, daily, such that copies of all data stored in storage system 104 during this period may be backed up within a single time step. This could be a computationally efficient method of data replication. Thus, in stage 403, processor 108 in storage system 104 may create a copy of one or more data items stored in memory 110 in one or more stages 401, and may send the copy data over network 107 to Backup storage system 105 . The processor 108 in the storage system 104 can communicate with the processor 112 in the backup storage system 105 so that the backup storage system 105 saves the copy data in the memory 113 . Stage 403 may include: the backup storage system 105 sends a report back to the storage system 104 to confirm the storage of the duplicate data.
在示例中,阶段403还可以包括:处理器108使得标识副本数据在备份存储系统105的存储器113中的位置的位置数据被存储。例如,参考图6,在示例中,阶段403可以包括步骤601的过程,即将位置数据与副本数据一起存储在存储器113中。因此,在需要从存储器中检索副本数据时,副本数据在存储器113中的位置可以,例如,由处理器108或112方便标识。In an example, stage 403 may also include the processor 108 causing location data identifying the location of the replica data in the memory 113 of the backup storage system 105 to be stored. For example, referring to FIG. 6 , in an example, stage 403 may include the process of step 601 , ie storing the location data in memory 113 together with the copy data. Accordingly, the location of the duplicate data in memory 113 may be conveniently identified, for example, by processor 108 or 112 when the duplicate data needs to be retrieved from memory.
在示例中,阶段403还可以包括:在存储系统104的存储器110中存储一个或多个标志,该标志标记计算机存储器110中存储在阶段403中复制的数据的位置的。因此,这些标志可以识别,例如,向处理器108,存储在存储器110中的已经被复制的部分数据,即备份副本已经存储在备份存储系统105中的数据。因此,在存储器110中的数据丢失时,处理器108能够确认备份存储系统105中是否存在该数据的副本,从而迅速检索副本数据。In an example, stage 403 may also include storing in memory 110 of storage system 104 one or more flags marking locations in computer memory 110 where data copied in stage 403 is stored. Thus, these flags may identify, for example, to processor 108 , portions of data stored in memory 110 that have been replicated, ie data for which a backup copy has been stored in backup storage system 105 . Therefore, when the data in the memory 110 is lost, the processor 108 can confirm whether there is a copy of the data in the backup storage system 105, so as to quickly retrieve the copy data.
在阶段404中,计算机程序201可以使得处理器108根据在阶段403中将数据的副本存储在备份存储中的结果,确认存储器110中用于在阶段402中存储相应数据的纠删码的位置可用于存储其它信息。换句话说,在阶段403中复制数据,使得数据的副本存储在备份系统105中,在此之后,存储系统104中的处理器108可以确定不再需要存储在存储器110中的数据的纠删码,而且纠删码在存储器中占用的位置可以被其它(例如,新)数据覆盖。例如,处理器108可以在存储器110中保留存储器110中可用于(即可以允许用于)存储其它数据的位置的寄存器。在示例中,这一阶段因此可以包括:处理器108修改该寄存器,以在寄存器中确认在阶段402中存储纠删码的位置可用于存储其它数据。因此,在稍后的时间步长中,当处理器108想要确认存储器110中用于存储其它数据的可用位置时,处理器可以查阅寄存器。在其它示例中,阶段404可以包括:处理器108在存储器110中存储纠删码的位置上设置标记,该标记用于确认可用于存储其它数据的存储位置。在示例中,阶段404可以包括:处理器108使得在阶段402中存储的纠删码从它们在存储器110中的相应位置上擦除。In stage 404, the computer program 201 may cause the processor 108 to confirm that the location in the memory 110 for storing the erasure code for the corresponding data in stage 402 is available as a result of storing a copy of the data in backup storage in stage 403 to store other information. In other words, the data is replicated in stage 403 so that a copy of the data is stored in the backup system 105, after which the processor 108 in the storage system 104 can determine that the erasure coding of the data stored in the memory 110 is no longer needed , and the location occupied by the erasure code in memory can be overwritten by other (eg, new) data. For example, processor 108 may reserve registers in memory 110 that are locations in memory 110 that are available (ie, may allow use) to store other data. In an example, this stage may thus include the processor 108 modifying the register to confirm in the register that the location where the erasure code was stored in stage 402 can be used to store other data. Thus, at a later time step, when processor 108 wishes to ascertain an available location in memory 110 for storing other data, the processor may consult the register. In other examples, stage 404 may include: the processor 108 sets a flag on the location where the erasure code is stored in the memory 110, and the flag is used to identify a storage location that can be used to store other data. In an example, stage 404 may include processor 108 causing erasure codes stored in stage 402 to be erased from their corresponding locations in memory 110 .
在示例中,阶段404可以包括参考图8描述的阶段801的过程。因此,阶段404可以包括只擦除一组纠删码(例如,奇偶校验“p”),同时即使在阶段403中复制数据之后,也将第二组纠删码(例如,奇偶校验“q”)保存在存储器110中。这可以有利地平衡最大程度减少内存占用空间和维持存储在存储器110中的数据的恢复能力这两个相互冲突的需求。In an example, stage 404 may comprise the process of stage 801 described with reference to FIG. 8 . Thus, stage 404 may include erasing only one set of erasure codes (e.g., parity "p"), while even after copying the data in stage 403, erasing a second set of erasure codes (e.g., parity "p") q") is stored in the memory 110. This may advantageously balance the conflicting needs of minimizing the memory footprint and maintaining the resiliency of data stored in memory 110 .
在示例中,阶段404可以包括参考图9描述的阶段901的过程。因此,阶段404可以包括又一步骤:将其它数据存储在计算机存储器110中在阶段404中确认可用于存储其它数据的位置上。In an example, stage 404 may comprise the process of stage 901 described with reference to FIG. 9 . Accordingly, stage 404 may comprise a further step of storing other data in the computer memory 110 at a location identified in stage 404 as available for storing the other data.
最后参考图10,在示例中,由处理器108运行的计算机程序201可以使得处理器108执行错误检测操作1001,由此处理器108检查存储在存储器110中的数据的完整性,以检查数据是否丢失或损坏。在处理器检测到数据丢失或损坏时,处理器108则可以从备份存储105的存储器113中检索丢失/损坏数据的副本。换句话说,在示例中,上述方法可以包括:测试数据以检查错误,例如,数据的部分或全部丢失或损坏,并且响应于检测到数据丢失/损坏,上述方法可以包括:检索副本数据。这种“主动”测试数据和检索副本数据的过程可以减少数据丢失和使用副本数据来恢复数据之间的时间,从而可以有利地最小化客户端在数据丢失后访问数据的时间。Referring finally to FIG. 10 , in an example, computer program 201 run by processor 108 may cause processor 108 to perform error detection operation 1001 whereby processor 108 checks the integrity of data stored in memory 110 to check whether the data lost or damaged. When the processor detects data loss or corruption, the processor 108 may then retrieve a copy of the lost/corrupted data from the memory 113 of the backup storage 105 . In other words, in an example, the above method may include testing the data to check for errors, eg, partial or total loss or corruption of the data, and in response to detecting data loss/corruption, the above method may include retrieving duplicate data. This process of "proactively" testing data and retrieving replica data can reduce the time between data loss and using the replica data to restore the data, thereby advantageously minimizing the time for clients to access the data after data loss.
本公开的各个方面在本文中已经在计算系统的上下文中描述,该计算系统包括服务远程客户端设备102和103的主存储系统104和远离存储系统104并服务于存储系统104的备份存储系统105。然而,本公开的各个方面相比于这种配置,实用性更广。例如,在本公开的各个方面的简单示例中,计算系统101可以包括整体式设备,而且数据、一个或多个纠删码和副本数据都可以存储在该整体式设备的存储中。在示例中,数据、一个或多个纠删码和副本数据都可以存储在单个存储设备中,例如,存储在单个磁盘驱动器中。这种计算系统可以包括计算机存储,例如,一个或多个存储设备,例如,磁盘驱动器,还包括耦合到存储的处理器,该处理器用于将数据存储在计算机存储中,将数据的纠删码存储在计算机存储中,将数据的副本存储在计算机存储中,以及根据将数据的副本存储在计算机存储中的结果,确认存储数据的纠删码的计算机存储可用于存储其它数据。Various aspects of the present disclosure have been described herein in the context of a computing system that includes a primary storage system 104 serving remote client devices 102 and 103 and a backup storage system 105 remote from and serving storage system 104 . However, aspects of the present disclosure have broader applicability than this configuration. For example, in a simple example of various aspects of the present disclosure, computing system 101 may comprise a monolithic device, and data, one or more erasure codes, and replicated data may all be stored in the monolithic device's storage. In an example, the data, one or more erasure codes, and replica data may all be stored on a single storage device, eg, on a single disk drive. Such a computing system may include computer storage, e.g., one or more storage devices, e.g., disk drives, and a processor coupled to the storage for storing data in the computer storage, erasure coding the data storing in computer storage, storing a copy of data in computer storage, and based on the results of storing a copy of data in computer storage, confirming that erasure coded computer storage storing data can be used to store other data.
虽然已经详细描述了本公开的各个方面及其相关优点,但是应该理解,在不脱离所附权利要求书界定的奔赴买那个的精神和范围的情况下,可以在此做出各种改变、替代和更改。在权利要求书中,词语“包括”不排除其它元件或步骤,并且“一”或“一个”不排除多个。虽然本公开提供的方法的过程在本文中已经描述为按特定顺序发生,但是在本公开的其它示例中,这些方法的过程可以按替代顺序执行,或者甚至可以从方法中省略。Although various aspects of the present disclosure and their associated advantages have been described in detail, it should be understood that various changes, substitutions and substitutions can be made herein without departing from the spirit and scope of the invention as defined by the appended claims. and change. In the claims, the word "comprising" does not exclude other elements or steps, and "a" or "an" does not exclude a plurality. Although the procedures of the methods provided by this disclosure have been described herein as occurring in a particular order, in other examples of the disclosure, the procedures of these methods may be performed in an alternate order, or may even be omitted from the methods.
Claims (15)
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/EP2020/086478 WO2022128080A1 (en) | 2020-12-16 | 2020-12-16 | Storing data in computer storage |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116601609A true CN116601609A (en) | 2023-08-15 |
Family
ID=74106020
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202080107869.XA Pending CN116601609A (en) | 2020-12-16 | 2020-12-16 | store data in computer storage |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN116601609A (en) |
| WO (1) | WO2022128080A1 (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117873772B (en) * | 2023-12-14 | 2024-12-10 | 天翼云科技有限公司 | A data recovery method for EC replica hybrid redundant storage |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8799746B2 (en) * | 2012-06-13 | 2014-08-05 | Caringo, Inc. | Erasure coding and replication in storage clusters |
| US9286163B2 (en) * | 2013-01-14 | 2016-03-15 | International Business Machines Corporation | Data recovery scheme based on data backup status |
| US20190163374A1 (en) * | 2017-11-28 | 2019-05-30 | Entit Software Llc | Storing data objects using different redundancy schemes |
-
2020
- 2020-12-16 WO PCT/EP2020/086478 patent/WO2022128080A1/en not_active Ceased
- 2020-12-16 CN CN202080107869.XA patent/CN116601609A/en active Pending
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022128080A1 (en) | 2022-06-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10303570B2 (en) | Method and apparatus for managing data recovery of distributed storage system | |
| US9367394B2 (en) | Decoupled reliability groups | |
| CN106776130B (en) | A log recovery method, storage device and storage node | |
| US10031816B2 (en) | Systems and methods for healing images in deduplication storage | |
| CN106339276B (en) | A data recovery method and system based on data backup state | |
| CN110058787B (en) | Method, apparatus and computer program product for writing data | |
| WO2022142544A1 (en) | Method for preventing data loss from flash memory, solid state drive controller, solid state drive | |
| US10503620B1 (en) | Parity log with delta bitmap | |
| US10740189B2 (en) | Distributed storage system | |
| US8639968B2 (en) | Computing system reliability | |
| EP3794451B1 (en) | Parity log with by-pass | |
| US11487628B1 (en) | System and method for rapidly transferring and recovering large data sets | |
| US9286163B2 (en) | Data recovery scheme based on data backup status | |
| CN102629223B (en) | Method and device for data recovery | |
| US20230350753A1 (en) | Storage system and failure handling method | |
| CN116601609A (en) | store data in computer storage | |
| US11055190B1 (en) | System and method for facilitating storage system operation with global mapping to provide maintenance without a service interrupt | |
| US10585764B2 (en) | Data storage system comprising primary and secondary storage systems | |
| CN114610235B (en) | Distributed storage clusters, storage engines, two-replica storage methods and devices | |
| CN115878381A (en) | Data recovery method and device based on SRM disc, storage medium and electronic device | |
| US11494090B2 (en) | Systems and methods of maintaining fault tolerance for new writes in degraded erasure coded distributed storage | |
| US11928497B2 (en) | Implementing erasure coding with persistent memory | |
| US10747610B2 (en) | Leveraging distributed metadata to achieve file specific data scrubbing | |
| US12346196B2 (en) | Extended protection storage system PUT operation | |
| JP2009205571A (en) | Disk array system, disk array control method, and disk array control program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |