WO2015040711A1 - Storage device, method for controlling data in storage device, and storage system - Google Patents

Abstract

This storage device is provided with a recording device and a controller connected to the recording device and a computer. The controller receives a plurality of sub-blocks obtained on the basis of the splitting of a block of data from the computer and stores same in the recording device, and records storage information associating the block and the stored plurality of sub-blocks. The controller receives subject information indicating a sub-block within the block from the computer, and on the basis of the storage information, identifies the block to which the sub-block indicated by the subject information belongs, and on the basis of the storage information, generates redundant information indicating the stored plurality of sub-blocks and transmits same to the computer. The controller receives from the computer an updated sub-block that is a sub-block not indicated by the redundant information among the plurality of sub-blocks obtained by means of splitting the updated data of the block.

Description

Storage device, data control method in storage device, and storage system

The present invention relates to control of data for storing data in a storage device.

The computer system includes, for example, a storage device and a computer such as a host computer connected to the storage device via a communication network. The storage device includes, for example, a plurality of HDDs (Hard Disk Drives) as storage devices for storing data.

In order to reduce costs, the computer system executes a data amount reduction process when storing data in the storage device. Data amount reduction processing includes, for example, file compression processing (Compression) and deduplication processing (De-duplication).

フ ァ イ ル File compression processing reduces the data capacity by aggregating data segments with the same content in one file. On the other hand, deduplication processing reduces the data capacity by aggregating data segments of the same content detected not only within one file but also between files. Hereinafter, unit data as a unit of deduplication processing is referred to as “chunk”.

In Patent Documents 1 and 2, when content is stored in a storage device from a host computer via a communication network, the host computer determines whether each chunk is stored in the storage device in advance, and is not stored. It is described that only chunks are transmitted to the storage apparatus.

US Pat. No. 5,990,810 International Publication No. WO2012 / 101674

Duplicate determination may be performed by the host computer or the storage device.

For example, Patent Document 1 describes a method in which a storage apparatus performs duplication determination, the host computer acquires the duplication determination result from the storage apparatus, and stores only new chunks in the storage apparatus.

However, in the deduplication process of Patent Document 1, the host computer makes an inquiry for duplication determination to the storage apparatus for all chunks. For this reason, the host computer needs to transmit information necessary for duplication determination to the storage apparatus and receive the duplication determination result from the storage apparatus. Compared with a method in which duplication determination is performed only by the host computer, the performance is reduced by the round trip of the communication between the host computer and the storage apparatus.

Patent Document 2 describes a method of performing duplication determination by a host computer and transmitting only chunks not stored in the storage apparatus to the storage apparatus according to the result.

However, in the deduplication processing of Patent Document 2, it is necessary to save information used for duplication determination on all data stored in the storage device in the host computer. In such deduplication processing, for example, as the data to be stored increases, the size of information used for duplication determination also increases, and therefore the disk capacity of the host computer is compressed. In addition, since the number of pieces of information used for duplication determination increases and necessary information must be selected from among them, deduplication processing may become inefficient.

The storage apparatus includes a storage device and a controller connected to the storage device and the computer. The controller receives a plurality of sub-blocks obtained based on the division of the block of data from the computer and stores them in the storage device, and stores storage information that associates the blocks with the plurality of stored sub-blocks. The controller receives the target information indicating the sub-block in the block from the computer, identifies the block to which the sub-block indicated in the target information belongs based on the stored information, and stored based on the stored information Duplicate information indicating a plurality of sub-blocks is generated and transmitted to the computer. The controller receives, from the computer, an update subblock that is a subblock that is not indicated in the duplication information among a plurality of subblocks obtained by dividing the update data of the block.

FIG. 1 illustrates a storage system according to an embodiment. FIG. 2 shows a configuration example of software according to the embodiment. FIG. 3 shows an example of the chunk index table 1300. FIG. 4 shows an example of the duplicate information list 1400. FIG. 5 shows an example of the content management table 1500. FIG. 6 shows an example of the chunk data set index table 1600. FIG. 7 is a schematic diagram for explaining deduplication processing according to the embodiment. FIG. 8 shows an example of a data structure in the storage system. FIG. 9 illustrates an example of backup processing according to the embodiment. FIG. 10 shows details of the information creation process.

An example will be described below.

In the following description, various types of information may be described using the expression “xxx table”, but the various types of information may be expressed using a data structure other than a table. In order to show that it does not depend on the data structure, the “xxx table” can be called “xxx information”.

In the following description, “program” and “software” may be used as the subject. However, since the program and software are executed by the processor to perform processing determined using the memory and the communication port, the processor is used. It may be a subject explanation.

Further, the management computer may be composed of one or more computers.

In the description of the present embodiment, the logically collected data recognized from the host device with respect to the backup data is referred to as “content”. In addition to normal files, the contents include files that are a collection of normal files such as archive files, backup files, and virtual volume files. In addition, a data segment that is a unit of deduplication processing in content is called “chunk”. A format for storing several chunks in the storage device is called “chunk data set”.

One chunk data set is created so that chunks that are highly related to each other are aggregated. For example, by setting a predetermined number of chunks or data capacity in advance for each chunk data set and collecting chunks generated from content until the chunk data set is full, the chunk data set considering the locality of the data Can be generated. In other words, data is stored in the chunk data set regardless of the content delimiter, but if the chunk data set storing the first chunk of a certain content is specified, the same chunk data is also applied to subsequent chunks. It is likely that it can be obtained from the set. However, although the content stored by one process is highly likely to be stored in the same chunk data set, the difference data resulting from the subsequent update is highly likely to be stored in a different chunk data set. That is, as the number of updates increases, there is a higher possibility that chunks constituting one content are distributed to a plurality of chunk data sets.

Generally, the size of one chunk is several kilobytes or more. For this reason, when the duplication determination process is executed, if the chunks are compared in order from the top of the chunk, a lot of processing time and cost are required. Therefore, in the storage apparatus according to the present embodiment, it is possible to execute the duplicate determination process in a short time and at a low cost by using a chunk message digest. Message digest is a technique for outputting a fixed-length digest in response to an arbitrary length of data input. In this specification, the output result of the message digest is referred to as a fingerprint (hereinafter referred to as FP) or a hash value. The FP is a hash value calculated using an arbitrary hash function. As this hash function, it is preferable to use a hash function such as SHA256, which has extremely high randomness and is highly likely to have a unique hash value for each chunk. Moreover, it is sufficient to replace the FP with information that can identify the chunk, such as the chunk itself or another identifier.

FIG. 1 shows a storage physical configuration example according to the embodiment.

The storage is composed of a host computer 10 and a storage device 30. The host computer 10 and the storage device 30 are connected to be communicable via a predetermined communication network (for example, SAN (Storage Area Network) 2).

The host computer 10 has a communication interface, a storage device, and a processor connected to them. An example of the processor is a CPU (Central Processing Unit) 11. An example of the storage device is a memory 12. An example of the communication interface is an HBA (Host Bus Adapter) 13.

The memory 12 stores information and programs for managing the storage device 30. The CPU 11 implements various functions by executing a program stored in the memory 12.

The HBA 13 is an interface device for connecting the host computer 10 and each node 31. The node 31 in this embodiment is a storage controller.

The storage device 30 includes a storage controller 31 and a storage unit 32. The storage unit 32 includes a plurality of storage devices 321.

The storage controller 31 has a communication interface, a storage device, and a processor connected to them. Examples of communication interfaces include FE-I / F (Front End Inter / Face) 314 and BE-I / F (Back End Inter / Face) 315. Examples of the storage device include a control memory 312 and a cache memory 313. An example of the processor is a CPU 311.

The control memory 312 stores information and a program for controlling the storage unit 32. The CPU 311 realizes various functions by executing a program stored in the control memory 312. The cache memory 313 temporarily stores data written to the storage unit 32 and data read from the storage unit 32.

The FE-I / F 314 is an interface device for communicating with an external device existing at the front end of the host computer 10 or the like. The BE-I / F 315 is an interface device for the storage controller 31 to communicate with a storage device in the storage unit 32.

The storage unit 32 includes a plurality of HDDs (Hard Disk Drives) 321. Instead of the HDD 321, another type of storage device (nonvolatile storage device)), for example, an FM (Flash Memory) device such as an SSD (Solid State Device) may be employed. The storage unit 32 may have different types of storage devices. Further, an RG (RAID group) may be configured by a plurality of storage devices of the same type. The RG stores data according to a predetermined RAID level. In this embodiment, the RG is composed of a plurality of HDDs 321.

FIG. 2 shows a configuration example of software according to the embodiment.

The host computer 10 stores the primary deduplication processing software 121, the duplication information list 1400A, and the content management table 1500A in the memory 12.

The primary deduplication processing software 121 executes data backup processing in accordance with an instruction from a user or a predetermined schedule. The duplicate information list 1400A is created by the storage apparatus 30 (specifically, the secondary deduplication processing software 3121) and transmitted from the storage apparatus 30. The content management table 1500A is created by the host computer 10 (specifically, the primary deduplication processing software 121) based on the duplication information list 1400A. Details of the duplicate information list 1400A and the content management table 1500A will be described later.

The storage device 30 stores the secondary deduplication processing software 3121, the file system management unit 3122, the chunk index table 1300, the duplicate information list 1400B, the content management table 1500B, and the chunk data set index table 1600 in the control memory 312. Yes.

The secondary deduplication processing software 3121 performs deduplication processing on the data transmitted from the primary deduplication processing software 121 on the host computer 10 side. The file system management unit 3122 manages data stored in the storage unit 32.

The chunk index table 1300, the duplicate information list 1400, and the chunk data set index table 1600 are created by the storage device 30 (specifically, the secondary deduplication processing software 3121). The duplicate information list 1400B is created by the storage apparatus 30 based on the chunk index table 1300 and the content management table 1500B.

The content management table 1500B is created by the host computer 10 (specifically, the primary deduplication processing software 121) and transmitted from the host computer 10.

Note that the storage apparatus 30 stores a content management table corresponding to all contents stored in the storage apparatus 30 in the control memory 31. On the other hand, the host computer 10 temporarily stores only the content management table corresponding to the content to be backed up. Therefore, the content management table stored in the memory 12 of the host computer 10 is the content management table 1500A, and the content management table stored in the control memory 31 of the storage device 30 is the content management table 1500B. did.

Details of the chunk index table 1300, the duplicate information list 1400, the content management table 1500 (A, B), and the chunk data set index table 1600 will be described later.

FIG. 3 shows an example of the chunk index table 1300.

The chunk index table 1300 is created by the storage apparatus 30 based on an instruction from the host computer 10 (details will be described later). The chunk index table 1300 is stored in the control memory 312 of the controller 31 included in the storage device 30.

The chunk index table 1300 is used for the storage apparatus 30 to specify the content including the chunk with respect to the chunk transmitted from the host computer 10.

The chunk index table 1300 has an entry for each chunk. Each entry stores a hash value 1301, a content ID 1302, and an offset 1303 in association with each other.

In an entry of a certain chunk, the hash value 1301 indicates the hash value of the chunk. The content ID 1302 is information for identifying the content including the chunk. An offset 1303 indicates the position of the chunk in the content.

FIG. 4 shows an example of the duplicate information list 1400.

Since the data structures of the duplicate information lists 1400A and 1400B are the same, here, description will be made without distinguishing between the two tables. Hereinafter, when the description is made without distinguishing between the duplicate information lists, the duplicate information list 1400 may be described. The duplicate information list 1400 is created by the storage apparatus 30 (details will be described later). The duplicate information list 1400 is stored in the control memory 312 of the controller 31 and the memory 12 of the host computer 10 that the storage apparatus 30 has.

The duplication information list 1400 is created by the storage device 30 in order to cause the host computer 10 to perform part of the duplication determination process, and is transmitted from the storage device 30 to the host computer 10. The duplicate information list 1400 is created for each content. In the following description, content refers to an archive file including a plurality of files.

The duplicate information list 1400 has an entry for each chunk. Each entry has a hash value 1401, a chunk length 1402, and a chunk data set ID 1403.

In an entry of a certain chunk, a hash value 1401 indicates the hash value of the chunk. The chunk length 1402 indicates the data length of the chunk. The chunk data set ID 1403 is information for identifying the chunk data set in which the chunk is stored.

FIG. 5 shows an example of the content management table 1500.

Since the data structures of the content management tables 1500A and 1500B are the same, here, description will be made without distinguishing between the two tables. Hereinafter, when the two content management tables are described without distinction, the content management table 1500 may be described.

The content management table 1500 associates the chunk stored for each chunk data set with the content. For example, when reading content in response to a read request from the host computer 10, the storage apparatus 30 specifies and reads a chunk data set that stores chunks included in the content based on the content management table 1500.

The content management table 1500 is stored in the control memory 312 of the controller 31 and the memory 12 of the host computer 10 that the storage apparatus 30 has.

The storage device 30 creates a content management table 1500B for each content. The content management table 1500B of a certain content has a file name including the content ID of the content, and has an entry for each chunk in the content. Each entry stores a content ID 1501, a hash value 1502, a chunk data set ID 1503, a content offset 1504, and a chunk length 1505 in association with each other.

In the entry of one chunk in specific content, the content ID 1501 is information for identifying the content. A hash value 1502 indicates a hash value of the chunk. The chunk data set ID 1503 is information for identifying the chunk data set in which the chunk is stored. The content offset 1504 indicates the position of the chunk in specific content. The chunk length 1505 indicates the data length of the chunk.

FIG. 6 shows an example of the chunk data set index table 1600.

Storage device 30 manages chunks for each chunk data set. That is, the chunk data set index table 1600 is used by the storage apparatus 30 to manage chunks for each chunk data set. The chunk data set index table 1600 is stored in the control memory 312 of the controller 31 included in the storage device 30.

The storage apparatus 30 creates a chunk data set index table 1600 for each chunk data set. The chunk data set index table 1600 of a certain chunk data set has a file name including the chunk data set ID of the chunk data set, and has an entry for each chunk in the chunk data set. Each entry has a hash value 1601, a chunk data set offset 1602, and a chunk length 1603.

In an entry of a certain chunk in a certain chunk data set, a hash value 1601 indicates the hash value of the chunk. The chunk data set offset 1602 indicates the position of the chunk in the chunk data set. The chunk length 1603 indicates the data length of the chunk.

FIG. 7 is a schematic diagram for explaining an overview of the deduplication processing according to the embodiment.

The outline of the deduplication process will be described in the order of the processes S1 to S14. It is assumed that the storage system is in the following state before the processes S1 to S14 are performed.
(1) The storage device 30 stores a chunk group “a, b, c, d, e” generated from the pre-update content F1.
(2) The representative chunk of the chunk group “a, b, c, d, e” is “a”.
(3) The chunk index table 1300 including the representative chunk “a” is stored in the control memory 31 of the storage device 30.
(4) A content management table 1500 related to the content F 1 is created by the host computer 10 and stored in the control memory 31 of the storage device 30.
(5) On the host computer 10 side, the updated content of the content F1 is input, and the chunk group generated from the updated content is “a, b, c, d1, e”.

<Process S1>
In the host computer 10, for example, content to be backed up is designated by the user. Here, the content F1 is designated by the user.

The host computer 10 divides the update content F1 into a plurality of data segments. Further, the host computer 10 divides the update content F1 into a plurality of unit data. Here, for example, the host computer 10 recognizes unit data by dividing the content from the top into predetermined data sizes (for example, 64 MB). When the content includes a plurality of files, the unit data may be a file. Further, both the delimiter for each predetermined data size and the delimiter for the file may be used as a unit data delimiter. Further, the size of the data segment is not a fixed size, and may be variable according to the data characteristics.

The host computer 10 generates a chunk by compressing the data segment, and determines a chunk (hereinafter, representative chunk) corresponding to the data segment representing each unit data. The representative chunk can be, for example, a chunk corresponding to the first data segment after the division when the content is divided into unit data. In this example, the representative chunk is “a”. Further, the host computer 10 calculates the hash values of all the generated chunks as the first calculated hash value.

<Process S2>
At this stage, the duplicate information list 1400A is empty. The host computer 10 refers to the duplicate information list 1400A and recognizes that the duplicate information list 1400A is empty.

In the processing flow, the contents of the duplicate information list 1400 stored in the host computer 10 and the storage device 30 may be different. Therefore, in the following description, the duplicate information list stored only in the host computer 10 is referred to as the duplicate information list 1400A, and the duplicate information list 1400A updated by the storage apparatus 30 is represented as the duplicate information list 1400B. is there.

<Process S3>
When the content of the duplicate information list 1400 </ b> A is empty, the host computer 10 transmits the representative chunk to the storage device 30.

<Process S4>
The storage apparatus 30 calculates the hash value of the chunk received from the host computer 10 as the second calculated hash value, and determines whether the second calculated hash value exists in the chunk index table 1300. If the chunk index table 1300 exists, the storage apparatus 30 refers to the chunk index table 1300 to identify the content ID 1302 corresponding to the hash value 1301 that matches the second calculated hash value. In this example, the specified content ID indicates the content F1. That is, the received chunk a is a representative chunk of the content F1 stored in the storage device 30.

<Process S5>
The storage device 30 identifies the content management table 1500B corresponding to the identified content ID, and refers to the identified content management table 1500B to identify all chunks included in the identified content.

The storage apparatus 30 creates a duplicate information list 1400B including the hash value 1502 of the identified chunk, and transmits the created duplicate information list 1400B to the host computer 10. The duplicate information list 1400B is created for each content. In this example, the duplicate information list 1400B includes hash values of “a, b, c, d, e”. The deduplication list 1400B to be transmitted includes the hash values of “b, c, d, e” and does not have to include the hash value of “a” that has already been received by the storage apparatus.

<Processing S6 and S7>
The host computer 10 uses the received duplicate information list 1400B to determine whether a chunk other than the representative chunk that constitutes the update content exists in the duplicate information list. Specifically, the host computer 10 determines whether or not the first calculation hash value of a chunk other than the representative chunk exists in the duplicate information list 1400B. In the following description, the hash value of the chunk calculated by the host computer 10 is referred to as a first calculated hash value, and the hash value of the chunk calculated by the storage device 30 is referred to as a second calculated hash value. The functions used for calculating the first calculated hash value and the second calculated hash value are the same.

Here, the host computer 10 determines that the hash values “b” and “c” exist in the duplicate information list 1400B. Chunks “b” and “c” having hash values in the duplicate information list 1400 are not transmitted to the storage apparatus 30 because the chunks exist in the storage apparatus 30.

<Process S8>
The host computer 10 determines whether or not the hash values of the remaining chunks that constitute the update content exist in the duplicate information list 1400B. Here, the host computer 10 determines that the hash value of the chunk “d1” does not exist in the duplicate information list 1400A.

<Processing S9 and S10>
Since the hash value does not exist in the duplicate information list 1400B, the host computer 10 transmits the chunk “d1” that has a high possibility that no chunk exists in the storage apparatus 30 to the storage apparatus 30. The storage apparatus 30 calculates the hash value of “d1” to obtain the second calculated hash value, and determines whether or not it exists in the chunk index table 1300. Since the second calculated hash value of “d1” is not the hash value of the representative chunk, it does not exist in the chunk index table 1300. Therefore, the storage apparatus 30 refers to the chunk data set index table 1600 and determines whether or not the second calculated hash value “d1” exists. If not, the storage apparatus 30 stores the second calculated hash value of “d1” in the chunk data set index table 1600.

<Process S11>
The storage device 30 generates a new chunk data set in the storage unit 32 and stores the chunk “d1” in the chunk data set.

<Process S12>
The storage apparatus 30 transmits the chunk data set ID of the chunk data set storing “d1” to the host computer 10. By receiving the chunk data set ID transmitted from the storage device 30, the host computer 10 can create a content management table 1500A related to the content including the received chunk data set ID.

<Process S13>
The host computer 10 determines whether or not the hash values of the remaining chunks that constitute the update content exist in the duplicate information list 1400B. Here, the host computer 10 determines that “e” exists in the duplicate information list 1400B. The “e” existing in the duplicate information list 1400B is not transmitted to the storage device 30.

<Process S14>
The host computer 10 performs duplication determination processing on all the chunks constituting the update content, and then generates the content management table 1500A for the update content based on the information received from the storage device 30, and the generated content The management table 1500A is transmitted to the storage device 30. In this example, the generated content management table 1500A indicates that the updated content includes the chunk group “a, b, c, d1, e”.

The storage device 30 receives the content management table 1500A of the updated content, and replaces the content management table 1500B of the content before update stored in the control memory 31 with the received content management table 1500A. By replacing the content management table 1500 with the one before and after the update, the deduplication list 1400 based on the latest content data can be created in the next deduplication process. In other words, the match rate between the chunks constituting the new update content of the host computer 10 and the chunks included in the latest deduplication list 1400 transmitted from the storage apparatus 30 increases, and the host apparatus 10 transfers to the storage apparatus 30. And the number of chunks to be sent can be reduced.

In the above description, it is assumed that the representative chunk “a” has not been updated. However, the representative chunk “a” may be updated (for example, updated to “a1”). In that case, the storage apparatus 30 newly associates the updated representative chunk “a1” with the new content ID 1302, registers it in the chunk index table 1300, and generates a new content management table 1500B corresponding to the content ID. (Refer to the dotted frame in FIG. 7).

Next, backup processing using the above-described deduplication processing will be described.

FIG. 9 shows an example of backup processing according to the embodiment.

The primary deduplication processing software 121 (hereinafter, software 121) of the host computer 10 divides the content to be backed up into data segments (S901).

The software 121 sequentially selects the divided data segments, and executes S902 to S909 for the selected data segments.

The software 121 generates a chunk by compressing the selected data segment and sets it as a target chunk (S902). The software 121 acquires the length of the target chunk, further calculates the hash value of the target chunk, and sets it as the first calculated hash value (S903).

The software 121 determines whether or not the first calculated hash value exists in the duplicate information list 1400A based on the length of the target chunk acquired in S903, the first calculated hash value of the target chunk, and the like (S904). ).

When the first calculated hash value exists in the duplicate information list 1400A (S904: Yes), the software 121 acquires the chunk data set ID 1403 associated with the first calculated hash value from the duplicate information list 1400A (S905). ).

The software 121 sets the hash value of the target chunk, the chunk data set ID associated with the hash value, and the length of the target chunk to the hash value 1502, the chunk data set ID 1503, and the chunk length 1505 in the content management table 1500. Each is stored (S909).

When the processing of S902 to S909 is completed for all the data segments, the software 121 transmits the content management table 1500A to the storage device 30 (S910). The storage apparatus 30 receives the content management table 1500A from the host computer 10, and replaces the content management table 1500B of the backup target content in the control memory 312 with the received content management table A (S911).

On the other hand, when the first calculated hash value does not exist in the duplicate information list 1400A (S904: No), the software 121 transmits the target chunk to the storage device 30 (S906).

The storage apparatus 30 performs information creation processing for creating the duplicate information list 1400B and the chunk index table 1300 based on the target chunk transmitted from the software 121 in S906. Details of this processing will be described with reference to FIG. 10 (S907).

The software 121 receives the chunk data set ID of the target chunk from the storage device 30 (S908). Thereafter, the software 121 performs the processing after S909.

According to the above backup processing, in the communication when updating the content stored in the storage device 30, the host computer 10 is determined not to overlap with the representative chunk of the content to be backed up in the content. The chunk and the content management table 1500A of the content are transmitted to the storage device 30. On the other hand, the storage device 30 transmits to the host computer 10 the deduplication list 1400A indicating the chunks in the content and the chunk data set ID of the chunks determined not to be duplicated in the content. Thereby, the storage capacity required for the duplication determination in the host computer 10 can be suppressed, and the load on the network between the host computer 10 and the storage device 30 can be suppressed.

FIG. 10 shows details of the information creation process.

The information creation processing is performed by secondary deduplication processing software 3121 (hereinafter, software 3121) stored in the control memory 31 of the storage device 30.

The software 3121 calculates the hash value of the target chunk transmitted from the software 121 in S906 as a second calculated hash value, and determines whether the second calculated hash value exists in the hash value 1301 of the chunk index table 1300. Determination is made (S1001).

When the second calculated hash value exists in the chunk index table 1300 (S1001: Yes), the software 3121 acquires the content ID 1302 and the offset 1303 corresponding to the second calculated hash value from the chunk index table 1300 ( S1002).

The software 3121 refers to the content management table 1500 corresponding to the content ID 1302 acquired in S1002 (S1003). The software 3121 acquires the hash value 1502, the chunk data set ID 1503, and the chunk length 1505 of each chunk constituting the target content of the acquired content ID 1302 from the content management table 1500B specified in S1003.

The software 3121 creates a duplicate information list 1400B including the information acquired in S1003 (S1005). The software 3121 transmits the duplicate information list 1400B created in S1005 to the host computer 10 (S1006).

The software 3121 determines whether the target chunk is a representative chunk (S1010). When transmitting a chunk to the storage apparatus 30, the host computer 10 gives information indicating whether or not the chunk is a representative chunk. The software 3121 determines that the transmitted target chunk is the representative chunk when information representing the representative chunk is given.

When the target chunk is the representative chunk (S1010: Yes), the software 3121 stores the second calculated hash value, the content ID, and the offset of the target chunk in the chunk index table 1300. On the other hand, when the target chunk is not the representative chunk (S1010: No), the software 3121 ends the process of S907.

When the second calculated hash value does not exist in the chunk index table 1300 (S1001: No), the software 3121 determines whether the target chunk is duplicated (S1007). Details of the processing of S1007 will be described later.

If the target chunks are not duplicated (S1007: No), the software 3121 generates a chunk data set index table 1600 of a new chunk data set, and registers the target chunk in the generated chunk data set index table 1600. Here, the software 3121 stores the hash value 1601 of the target chunk, the chunk data set offset 1602, and the chunk length 1603 in the generated chunk data set index table 1600 (S1008).

On the other hand, when the chunks overlap (S1007: Yes), the software 3121 refers to the content ID 1501 of the content management table 1500B and transmits the chunk data set ID in which the target chunk is stored to the host computer 10 ( S1009). Thereafter, the software 3121 performs the processing after S1010.

According to the information creation process described above, the storage apparatus 30 can transmit information indicating the content and chunk data set corresponding to the target chunk received from the host computer 30 to the host computer 30.

On the other hand, instead of transmitting the deduplication list 1400 from the storage apparatus 30 to the host computer 10, it is possible to transmit the chunk data set index table 1600 as it is.

However, in the storage device 30, when the chunks that make up the content are updated, the updated chunks are stored in a new chunk data set, and a new chunk data set index table 1600 is created. The chunk data set index table 1600 is not updated. Therefore, by repeatedly updating the content, chunks constituting the latest content are distributed and stored in a plurality of chunk data sets, and even if the chunk data set index table 1600 is transmitted to the host computer 10, In the host computer 10, the probability that it can be determined that the chunks are duplicated decreases.

However, with the above information processing, the storage apparatus 30 can create a deduplication list 1400 having information on the chunks that make up the same latest content as the chunk transmitted from the host computer 10 and transmit it to the host computer 10. it can. Thereby, it is possible to increase the probability that the host computer 10 can determine that the chunks are duplicated.

Next, the details of S1007 will be shown.

FIG. 8 shows an example of the data structure in the storage system.

In this figure, elements having the same reference numerals as those in FIG. 2 are the same as those in FIG. The chunk index (Chunk Index) 1200 stores the hash values of all the chunks stored in the storage device 30 in association with the chunk data set ID of the chunk data set in which the chunk is stored.

Content 1700 has an arbitrary file name and has several chunks of data. A chunk data set 1800 has a file name including a chunk data set ID, and has a length and data of several chunks.

The software 3121 uses the chunk index 1200 to identify the chunk data set ID associated with the hash value of the target chunk. Thereafter, the software 3121 refers to the chunk data set index table 1600 corresponding to the specified chunk data set ID, and determines whether or not the target chunk is stored in the storage device 30 from the hash value and the length. If the second calculated hash value and the length of the target chunk match, the software 3121 determines that the chunk is duplicated.

The host computer 10 may acquire the user ID of the user of the content to be backed up from the content and send it to the storage device 30 together with the representative chunk. In this case, the storage device 30 associates the user ID with the content ID, and recognizes that the content is different from each other if the data of the two contents is the same, but is associated with different user IDs. Thereby, the storage apparatus 30 can distinguish and manage the contents of different users.

The chunk is a data segment compressed, but may be the data segment itself. In the above embodiment, the storage system in which the two cases of the host computer 10 and the storage device 30 cooperate is described. However, the storage system has the processor of the host computer 10 and the storage controller 31 in one case. Also good.

In the storage device of the present invention, the computer corresponds to the host computer 10 or a backup server device. The storage device corresponds to the storage device 321 and the like. A block corresponds to content or the like. A sub-block corresponds to a chunk or a data segment. The storage information corresponds to the chunk index table 1300, the content management table 1500, and the like. The sub-block information corresponds to the chunk index table 1300 and the like. The block information corresponds to the content management table 1500B and the like. The duplicate information corresponds to the duplicate information list 1400 and the like. The storage location corresponds to a chunk data set or the like. The position information corresponds to the chunk data set index table 1600 and the like. The update information corresponds to the content management table 1500A and the like. Some of the sub-blocks correspond to representative chunks. The representative sub block corresponds to a representative chunk or the like. The first controller corresponds to the storage controller 31 and the like. The second controller corresponds to the CPU 11 and the like.

Although several embodiments have been described above, these embodiments are merely examples, and the present invention can be applied to various other modes.

DESCRIPTION OF SYMBOLS 10 ... Host computer, 30 ... Storage apparatus, 1300 ... Chunk index table, 1400 ... Duplicate information list, 1500 ... Content management table, 1600 ... Chunk data set index table

Claims (10)

A storage device;
A controller connected to the storage device and a computer,
The controller is
A plurality of sub-blocks obtained based on the division of the block of data are received from the computer and stored in the storage device;
Storing storage information associating the block with the plurality of stored sub-blocks;
Receiving target information indicating sub-blocks in the block from the computer;
Based on the storage information, identify a block to which the sub-block indicated in the target information belongs,
Based on the stored information, generate duplicate information indicating the plurality of stored sub-blocks and send to the computer,
Receiving an update sub-block from the computer, which is a sub-block not shown in the duplication information among a plurality of sub-blocks obtained by dividing the update data of the block;
Storage device. The storage device according to claim 1,
The storage information includes a part of the plurality of stored sub-blocks, sub-block information indicating an association with the block, the block, the plurality of stored sub-blocks, Block information indicating an association with storage positions of a plurality of stored sub-blocks,
The controller is
Based on the sub-block information, identify a block to which the sub-block indicated in the target information belongs,
Based on the block information, the duplicate information indicating the stored sub-blocks is generated.
Storage device. The storage device according to claim 2,
The controller is
Storing position information indicating each storage position of the plurality of stored sub-blocks;
Based on the position information, generating the duplication information including the storage position of each of the stored sub-blocks,
In response to receiving the update subblock, based on the location information, determine whether the update subblock is stored in the storage device;
If it is determined that the updated sub-block is not stored in the storage device, the updated sub-block is stored in the storage device;
Storage device. The storage device according to claim 3,
The controller is
Transmitting the storage location of the update sub-block in the storage device to the computer;
The update information based on the duplication information and the storage location of the update sub-block, the update information indicating the storage location of all the sub-blocks included in the update data is received from the computer,
Updating the block information with the update information;
Storage device. The storage apparatus according to claim 4, wherein
The target information is a representative sub-block selected by a predetermined rule from a plurality of sub-blocks included in the update data,
The sub-block information indicates an association between the representative sub-block and the block.
Storage device. The storage device according to claim 5,
The controller calculates a message digest of each of the stored sub-blocks and includes it in the block information,
The duplicate information includes a message digest of each of the stored sub-blocks.
Storage device. The storage apparatus according to claim 6, wherein
The plurality of sub-blocks are obtained by compressing a plurality of data segments obtained by dividing the block,
Storage device. The storage apparatus according to claim 7, wherein
The block information indicates an association between the block and a user of the block;
The controller recognizes a plurality of blocks associated with different users as different blocks.
Storage device. A storage device;
A first controller connected to the storage device;
A second controller connected to the first controller;
With
The second controller generates a plurality of sub-blocks based on the division of the block of data, and transmits the generated plurality of sub-blocks to the first controller;
The first controller is
Receiving the plurality of sub-blocks transmitted from the second controller, storing the received sub-blocks in the storage device;
Storing storage information associating the block with the plurality of stored sub-blocks;
The second controller transmits target information indicating a sub-block in the block to the first controller,
The first controller is
Receiving the target information transmitted from the second controller;
Based on the storage information, identify a block to which the sub-block indicated in the target information belongs,
Based on the storage information, generate duplicate information indicating the plurality of stored sub-blocks and send it to the second controller,
The second controller is a sub-block that generates a plurality of sub-blocks based on the division of the update data of the block and is not indicated in the duplication information among the plurality of sub-blocks generated from the update data Sending an update sub-block to the first controller;
The first controller receives the update sub-block transmitted from the second controller;
Storage system. A method for controlling data in a storage apparatus connected to a computer and managing storage devices,
The storage device
A plurality of sub-blocks obtained based on the division of the block of data are received from the computer and stored in the storage device;
Storing storage information associating the block with the plurality of stored sub-blocks;
Receiving target information indicating sub-blocks in the block from the computer;
Based on the storage information, identify a block to which the sub-block indicated in the target information belongs,
Based on the stored information, generate duplicate information indicating the plurality of stored sub-blocks and send to the computer,
Receiving an update sub-block from the computer, which is a sub-block not shown in the duplication information among a plurality of sub-blocks obtained by dividing the update data of the block;
A method for controlling data in a storage apparatus.

Images