CN106649676A - Duplication eliminating method and device based on HDFS storage file - Google Patents

Abstract

The embodiment of the invention discloses a duplication eliminating method and device based on an HDFS storage file. The method includes the steps that a file fingerprint of a file to be subjected to duplication eliminating is compared with a file fingerprint of a stored file; if the file fingerprints are the same, link identification is calculated according to file identification of the file to be subjected to duplication eliminating; the link identification and the storage address of the same storage file in a storage node replace the file content of the file to be subjected to duplication eliminating to serve as a key value of the file identification of the file to be subjected to duplication eliminating, and the key value is stored into the storage node. According to the duplication eliminating method and device in the technical scheme, files with the duplicated content are effectively removed, the number of files is decreased, the storage space is saved, and the system performance is improved.

Description

A method and device for deduplication based on HDFS storage files

technical field

Embodiments of the present invention relate to unstructured data storage technologies, and in particular to a deduplication method and device for storing files based on HDFS.

Background technique

Hadoop Distributed File System (HDFS for short) is a system that provides reliable storage for ultra-large-scale data sets. Provide high-bandwidth input and output data streams. HDFS has high fault tolerance and can run on low-cost hardware clusters. Using the master-slave architecture of MASTER/SLAVES, an HDFS cluster consists of a Namenode node (management node) and multiple Datanode nodes (storage nodes). The management node is a central server responsible for managing file system metadata and client access to files. The management node stores the metadata of the files, so the memory capacity of the management node limits the number of files. By default, HDFS divides files into blocks (storage blocks), for example, 64M is 1 storage block. Then store each storage block in the storage node of HDFS in the form of key-value pair, and store the mapping of key-value pair in memory. Each file, storage block, and index directory is stored in the form of an object in memory, and each object occupies about 150 bytes. For example, if there are 1,000,000 small files, and each file occupies a storage block, then the management node needs at least 300M of memory; if storing 100 million or more files, 20G or more memory capacity is required. The solution is to build an in-memory database that supports clusters, but this increases the cost of the system. If there are too many small files, excessive memory resources will be occupied and cluster performance will be affected. Small files need to be merged to reduce the number of files.

However, in actual Internet applications, there are a large number of small files, especially with the rise of social networking sites such as blogs, microblogs, and Facebook, which have changed the way the Internet stores content. Users have basically become the creators of Internet content, and their data has the characteristics of massive, diverse, and dynamic changes, resulting in a large number of small files, such as status files, user profiles, and avatars. According to the data storage format, these data can be divided into structured data and unstructured data. Structured data has the same hierarchical and grid structure and can be described by numbers or words; while some information cannot be represented by numbers or a uniform structure, such as scanned images, faxes, photographs, computer-generated reports, word processing Documents, spreadsheets, presentations, voice and video, these are unstructured data. After structured extraction of unstructured data, the original files need to be saved for subsequent use.

In many fields, the proportion of unstructured data is much higher than that of structured data. The amount of unstructured data information is very large. If it is directly stored in the database, in addition to greatly increasing the capacity of the database, it will also reduce the efficiency of maintenance and application. In particular, the unstructured data obtained on the Internet is often repetitive, and hot events will attract a large number of netizens' attention in a short period of time, resulting in a small amount of unstructured data being reused in a large amount in a short period of time, occupying system storage space. In the existing technology, compression technology is used to compress data according to a certain ratio, but unstructured data does not have a strict structure, and it is more difficult to standardize and manage than structured information. In response to these characteristics, the massive unstructured small files currently stored in HDFS are merged into large files using Mapfile technology, and they are not compressed, which takes up a lot of storage space. Therefore, how to remove duplicate content in massive unstructured data and save storage Space is an urgent issue.

Contents of the invention

Embodiments of the present invention provide a deduplication method and device based on HDFS stored files, so that HDFS can effectively deduplicate and save storage space when processing a large number of stored unstructured small files.

In the first aspect, the embodiment of the present invention provides a method for deduplication based on HDFS storage files, including:

Compare the file fingerprint of the file to be deduplicated with the file fingerprint of the stored file;

If the comparison result is the same, calculate the link identifier according to the file identifier of the file to be deduplicated;

Replace the file content of the file to be deduplicated with the link ID and the storage address of the same stored file in the storage node, and store it in the storage node as the key value of the file ID of the file to be deduplicated.

Preferably, before comparing the file fingerprint of the file to be deduplicated with the file fingerprint of the stored file, it also includes:

storing the received file in the set area in the storage node, and marking it as an area not deduplicated;

Obtain files one by one from the non-deduplication processing area as files to be deduplication.

Preferably, storing the received file in the setting area in the storage node includes:

Generate a primary key for the received file as a file identifier;

Converting the file content of the file into binary data, corresponding to the file identifier, and storing it in the setting area of the storage node.

Preferably, storing the received file in the setting area in the storage node includes:

According to the receiving date of the file, the received file is stored in different setting areas in the storage node.

Preferably, calculating the link identifier according to the file identifier of the file to be deduplicated comprises:

A 32-bit MD5 value is calculated for the file identifier of the file to be deduplicated as the link identifier.

Preferably, the file content of the file to be deduplicated is replaced with the link identifier and the storage address of the same stored file in the storage node, and stored in the storage node as the key value of the file identifier of the file to be deduplicated After that, also include:

Rewrite the index file of the storage node according to the storage location of each file identifier and the corresponding key value in the storage node.

Preferably, the method also includes:

Obtain the file identifier of the file to be read according to the received file read request;

Calculating a corresponding link identifier according to the file identifier;

Reading the set bit data of the corresponding key value from the storage node according to the file identifier;

If comparing the link identifier with the set bit data, read the storage address from the key value;

Locate and find the corresponding file in the storage node according to the storage address, and respond to the file read request after reading.

In the second aspect, the embodiment of the present invention also provides a deduplication device based on HDFS storage files, including:

The fingerprint comparison module is used to compare the file fingerprint of the file to be deduplicated with the file fingerprint of the stored file;

A link identification calculation module, configured to calculate a link identification according to the file identification of the file to be deduplicated if the comparison result is the same;

A content replacement module, configured to replace the file content of the file to be deduplicated with the link identifier and the storage address of the same stored file in the storage node, as the key value storage of the file identifier of the file to be deduplicated to the storage node.

Preferably, the device also includes:

The file storage module is used to store the received file in the set area in the storage node before comparing the file fingerprint of the file to be deduplicated with the file fingerprint of the stored file, and mark it as not deduplicated processing area;

The file acquisition module is configured to acquire files one by one from the non-deduplication processing area as files to be deduplication.

Preferably, the file storage module includes:

A primary key generating unit, configured to generate a primary key for the received file as a file identifier;

The content conversion unit is configured to convert the file content of the file into binary data, and store it in a setting area in the storage node corresponding to the file identifier.

Preferably, the file storage module is specifically used for:

According to the receiving date of the file, the received file is stored in different setting areas in the storage node.

Preferably, the link identification calculation module is specifically used for:

A 32-bit MD5 value is calculated for the file identifier of the file to be deduplicated as the link identifier.

Preferably, the device also includes:

The rewriting index module is used to replace the file content of the file to be deduplicated with the link identifier and the storage address of the same stored file in the storage node as the key value of the file identifier of the file to be deduplicated After being stored in the storage node, the index file of the storage node is rewritten according to the storage location of each file identifier and the corresponding key value in the storage node.

Preferably, the device also includes:

The file identification reading module is used to obtain the file identification of the file to be read according to the received file reading request;

A corresponding identification calculation module, configured to calculate a corresponding link identification according to the file identification;

The set bit data reading module is used to read the set bit data of the corresponding key value from the storage node according to the file identifier;

A matching module, configured to read the storage address from the key value if comparing the link identifier with the set bit data;

A file search module, configured to locate and find a corresponding file in the storage node according to the storage address, and respond to the file read request after reading.

The embodiment of the present invention is aimed at massive unstructured files with the same file content in HDFS, only one copy of the file with the same content is reserved, and the file content with the same fingerprint as the stored file is deleted and replaced with a link identifier and link address, effectively removing content duplication files, reduce the number of files, save a lot of storage space, release memory resources, improve system performance, and at the same time, meet the needs of fast storage and correct reading.

Description of drawings

FIG. 1A is a flow chart of a deduplication method based on HDFS storage files in Embodiment 1 of the present invention;

FIG. 1B is a schematic diagram of a deduplication method based on HDFS storage files in Embodiment 1 of the present invention;

Fig. 2 is a flow chart of a method for deduplication based on HDFS storage files in Embodiment 2 of the present invention;

Fig. 3 is a flow chart of a method for deduplication based on HDFS storage files in Embodiment 3 of the present invention;

4A is a schematic structural diagram of a deduplication device based on HDFS storage files in Embodiment 4 of the present invention;

FIG. 4B is a schematic structural diagram of an HDFS-based deduplication device for storing files in Embodiment 4 of the present invention.

detailed description

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, but not to limit the present invention. In addition, it should be noted that, for the convenience of description, only some structures related to the present invention are shown in the drawings but not all structures.

Embodiment one

FIG. 1A is a flow chart of a deduplication method based on HDFS storage files provided by Embodiment 1 of the present invention. This embodiment is applicable to Hadoop distributed file system, and the system generally includes a management node and multiple storage nodes. The method can be executed by a deduplication device based on HDFS storage files, the device can be implemented in software and/or hardware, and is generally integrated in a management node in a Hadoop distributed file system.

The method of Embodiment 1 of the present invention specifically includes:

S101. Compare the file fingerprint of the file to be deduplicated with the file fingerprint of the stored file.

The file to be deduplicated is a received file, the file can be stored in the storage node first, and then the deduplication operation of this embodiment is performed in an offline state, or the deduplication operation can be performed online when the deduplication file is received. Heavy operation. Since online deduplication requires large resources, slow operation speed, and long response time, it is preferable to perform offline deduplication. Extract files that have not been reprocessed in the past from the storage node as files to be deduplicated.

Specifically, the file fingerprint is calculated according to the content of each file. No matter how the file name changes, as long as the file content does not change, the calculated file fingerprint is the same. If the content of the file to be deduplicated is the same as that of the stored file, the calculated file fingerprints are the same. The calculation method of the file fingerprint can be the fifth edition of the message digest algorithm (Message-Digest Algorithm 5, referred to as MD5 value), the secure hash algorithm (Secure Hash Algorithm 1, referred to as SHA1 value) or cyclic redundancy check (Cyclic Redundancy) Check, CRC32 value for short). Among them, the MD5 value is highly discrete, and a small change in the original information content will lead to a huge change in the MD5 value, which is highly reliable. In this embodiment, it is preferable to obtain the first 1K binary data of the file and the last 1K binary data of the file to calculate the MD5 value, and the calculated result is used as the file fingerprint.

In the offline state, the file fingerprint of the file to be deduplicated is regularly compared with the file fingerprint of the stored file. After 0 o'clock every day, use the MapReduce computing model in the Hadoop distributed file system to compare the file fingerprints of the files to be deduplicated with the file fingerprints of the stored files offline, and filter out the files that have the same content as the stored files. Deduplicate files, and obtain the corresponding stored files and their storage addresses in the data storage nodes.

S102. If the comparison result is the same, calculate a link identifier according to the file identifier of the file to be deduplicated.

Specifically, when a file is written into the Hadoop distributed file system, it is stored in the form of a key-value pair Key-Value in the map file (Mapfile), and the primary key Key is the file identifier, which is assigned to the file when the file is stored A string that uniquely identifies the file. The key value Value is the binary value corresponding to the Key, that is, all the binary data corresponding to the file content. If the file fingerprint comparison results are the same, a link identifier is calculated according to the file identifier Key of the file to be deduplicated, and the link identifier plays a special identification role for the file that has been deduplicated. In the file reading stage, if the link identifier is read from the key value of the file instead of the actual binary data, it indicates that the file has been deduplicated. If the file fingerprint comparison results are not the same, it means that the file content of the file is different from that of the stored file, and the file content of the file is retained without deduplication processing.

Preferably, step S102 includes:

A 32-bit MD5 value is calculated for the file identifier of the file to be deduplicated as the link identifier.

In this embodiment, a 32-bit MD5 value is calculated according to the file identifier Key of the file to be deduplicated as the link identifier. Similar to the encryption process, the deduplicated file is encrypted and marked, and the 32-bit MD5 value is calculated, and decrypted when responding to the file read request. Moreover, at the stage of reading the file, the link identifier can be calculated according to the file identifier, so as to identify whether the file has been deduplicated.

S103. Replace the file content of the file to be deduplicated with the link identifier and the storage address of the same stored file in the storage node, and store it in the storage node as the key value of the file identifier of the file to be deduplicated. .

In this embodiment, for a file that undergoes deduplication processing, the binary data of the file content is no longer stored in the key value, but is replaced by a link identifier and a storage address. identical. As shown in Figure 1B, assuming that the content of the file corresponding to Key2 is the same as that of the stored file, the binary data corresponding to the content of the Key2 file is read out, and the link identifier and storage address are written in the original position, that is, the 32-bit MD5 value and the same file storage The actual address of the file to complete the replacement of the content of the file to be deduplicated. The contents of the files corresponding to Key1 and Key3 are different from the contents of the stored files, and the binary data corresponding to the contents of the files of Key1 and Key3 are reserved.

Preferably, step S103 includes:

Rewrite the index file of the storage node according to the storage location of each file identifier and the corresponding key value in the storage node.

Specifically, the data can be quickly located according to the index file. The key-value data corresponding to each file identifier in the storage node has been replaced, and the original index file cannot correctly represent the new mapping relationship. The storage location of each file identifier and the corresponding key value in the storage node, and rewrite the index file of the storage node.

Embodiment 1 of the present invention provides a deduplication method based on HDFS stored files, which compares file fingerprints and performs data deduplication processing in an offline state, which can appropriately extend the processing time, increase system reliability, and save memory resources. It reduces the requirements for hardware devices, thereby saving a lot of device costs, and can effectively remove files with duplicate content, reduce the number of files, and save storage space.

Embodiment two

Fig. 2 is a flow chart of a deduplication method based on HDFS storage files provided by Embodiment 2 of the present invention. Embodiment 2 of the present invention is optimized and improved on the basis of Embodiment 1, and how to perform offline deduplication operations is further explained. As shown in Figure 2, the second embodiment of the present invention specifically includes:

S201. Store the received file in a set area in the storage node, and mark it as an area not deduplicated.

In this embodiment, the Hadoop distributed file system includes multiple mapping files, and the mapping files are used for archiving a large number of unstructured small files and generating mapping relationships corresponding to the archive files. The system continuously receives files and caches them. When the cache space reaches the capacity threshold or the receiving time reaches the preset time limit, the system writes the unstructured files into the mapping files of each storage node in sequence according to the order in which they are received, and marks them as not deduplicated. Treatment area. Wherein, the range of the capacity threshold can be set between 128M and 2G, the range of the preset time limit can be set between 5 minutes and 20 minutes, and the writing method can be written concurrently through multiple threads to ensure that the writing speed.

Preferably, step S201 includes:

Generate a primary key for the received file as a file identifier;

Converting the file content of the file into binary data, corresponding to the file identifier, and storing it in the setting area of the storage node.

Specifically, the system generates a primary key Key for the received file as a file identifier, performs index storage according to the primary key Key, converts the file content of the file into binary data, and uses it as the key value corresponding to the primary key Key, the primary key Key and the corresponding The key-value Value is stored in the mapping file of the storage node in the form of key-value pairs.

Preferably, step S201 also includes:

According to the receiving date of the file, the received file is stored in different setting areas in the storage node.

Specifically, when storing a file, the received file is stored in different mapping files in the storage node according to the receiving date. For the files written every day, create a new directory in the Hadoop distributed file system for storage, and store them in partitions in units of days.

S202. Obtain files one by one from the non-deduplication processing area as files to be deduplication.

S203. Compare the file fingerprint of the file to be deduplicated with the file fingerprint of the stored file.

S204. If the comparison result is the same, calculate the link identifier according to the file identifier of the file to be deduplicated.

S205. Replace the file content of the file to be deduplicated with the link ID and the storage address of the same stored file in the storage node, and store it in the storage node as the key value of the file ID of the file to be deduplicated. .

Embodiment 2 of the present invention provides a deduplication method based on HDFS storage files, which stores files in partitions according to the acceptance date of the files, which is convenient for offline processing, and temporarily does not perform deduplication processing for the stored files of the day, which can ensure data storage Efficiency, meet the demand for fast data storage, and improve the real-time performance of data storage.

Embodiment three

Fig. 3 is a flow chart of a deduplication method based on HDFS storage files provided by Embodiment 3 of the present invention. Embodiment 3 of the present invention is optimized and improved on the basis of Embodiment 2. After the files are deduplicated, the obtained The process of the file content is further described, as shown in Figure 3, the third embodiment of the present invention specifically includes:

S301. According to the received file reading request, acquire the file identifier of the file to be read.

S302. Calculate a corresponding link identifier according to the file identifier.

S303. Read the set bit data of the corresponding key value from the storage node according to the file identifier.

S304. If comparing the link identifier with the set bit data, read the storage address from the key value.

S305. Locate and find the corresponding file in the storage node according to the storage address, and respond to the file read request after reading.

In this embodiment, the process of obtaining the file content shields the file user from the internal processing flow. The system obtains the file identification primary key Key of the file to be read according to the received file reading request, and calculates the key to be read according to the primary key Key. The link identifier corresponding to the file's primary key Key, which can be obtained by calculating the MD5 value. Read the first 32-bit MD5 value of the corresponding key value from the storage node according to the primary key Key, compare the MD5 value of the link identifier with the read first 32-bit MD5 value, if they are consistent, it means that the file to be read has passed Reprocessing, the file content stores the storage address of the stored file with the same content in the storage node, not the real content of the file itself, removes the first 32 bits of data in the file content, and reads the storage from the key value address, locate and find the corresponding file in the storage node according to the storage address, and respond to the file read request after reading. Compare the MD5 value of the link identifier with the first 32 MD5 values read, if they are inconsistent, it means that the file to be read has not been deduplicated, and respond to the file read after reading the file content from the key value ask.

Embodiment 3 of the present invention provides a deduplication method based on HDFS storage files. For repeated unstructured files, only the corresponding storage address is saved, and the internal processing flow is shielded from the visitor when reading the file, which can meet the requirements of correct reading. requirements, saving storage space and improving system performance.

Embodiment Four

FIG. 4A is a schematic structural diagram of a deduplication device based on HDFS storage files in Embodiment 4 of the present invention, and the device is applied to the Hadoop distributed file system. As shown in Figure 4A, the device includes:

Fingerprint comparison module 401, for comparing the file fingerprint of the file to be deduplicated with the file fingerprint of the stored file;

A link identification calculation module 402, configured to calculate a link identification according to the file identification of the file to be deduplicated if the comparison result is the same;

The content replacement module 403 is used to replace the file content of the file to be deduplicated with the link identifier and the storage address of the same stored file in the storage node as the key value of the file identifier of the file to be deduplicated stored in the storage node.

Preferably, the link identification calculation module is specifically used for:

A 32-bit MD5 value is calculated for the file identifier of the file to be deduplicated as the link identifier.

Preferably, the device also includes:

The rewriting index module 404 is used to replace the file content of the file to be deduplicated with the link identifier and the storage address of the same stored file in the storage node as the key of the file identifier of the file to be deduplicated After the value is stored in the storage node, the index file of the storage node is rewritten according to the storage location of each file identifier and the corresponding key value in the storage node.

Specifically, in the offline state, use the fingerprint comparison module to compare the file fingerprint of the file to be deduplicated with the file fingerprint of the stored file, filter out the file to be deduplicated with the same content as the stored file, and Obtain the corresponding stored file and its storage address in the data storage node. If the file fingerprint comparison result is the same, according to the file identification Key of the file to be deduplicated, calculate the 32-bit MD5 value in the link identification calculation module, and as the link identification, the link identification is used for the file that has been deduplicated. to the identification function. Through the content replacement module, the file content of the file to be deduplicated is replaced with the link identifier and the storage address of the same stored file in the storage node, and stored as the key value of the file identifier of the file to be deduplicated storage node. According to the storage location of each file identifier and the corresponding key value in the storage node, rewrite the index file of the storage node in the rewriting index module.

Preferably, as shown in Figure 4A, the device further includes:

The file storage module 405 is used to store the received file in the setting area in the storage node before comparing the file fingerprint of the file to be deduplicated with the file fingerprint of the stored file, and mark it as undeleted reprocessing area;

The file obtaining module 406 is configured to obtain files one by one from the non-deduplication processed area as files to be deduplicated.

Preferably, the file storage module includes:

A primary key generating unit, configured to generate a primary key for the received file as a file identifier;

The content conversion unit is configured to convert the file content of the file into binary data, and store it in a setting area in the storage node corresponding to the file identifier.

Preferably, the file storage module is specifically used for:

According to the receiving date of the file, the received file is stored in different setting areas in the storage node.

Specifically, the file access module continuously receives files and caches them. When the cache space reaches the capacity threshold or the receiving time reaches the preset time limit, the system writes them concurrently by multiple threads according to the acceptance date of the files and the order in which unstructured files are received. In the mapping file of each storage node. Wherein, the range of the capacity threshold can be set between 128M and 2G, and the range of the preset time limit can be set between 5 minutes and 20 minutes. The primary key generation unit generates the primary key Key for the received file, as the file identifier, utilizes the content conversion unit to convert the file content of the file into binary data, as the key value corresponding to the primary key Key, the primary key Key and the corresponding key value Value are The form of the key-value pair is stored in the mapping file of the storage node. According to the file acquisition module, the files are obtained one by one from the non-deduplication processing area as the files to be deduplication.

Preferably, as shown in Figure 4B, the device further includes:

The file identification reading module 407 is used to obtain the file identification of the file to be read according to the received file reading request;

A corresponding identification calculation module 408, configured to calculate a corresponding link identification according to the file identification;

The set bit data reading module 409 is used to read the set bit data of the corresponding key value from the storage node according to the file identifier;

A matching module 410, configured to read the storage address from the key value if comparing the link identifier matches the set bit data;

The file search module 411 is configured to locate and find a corresponding file in the storage node according to the storage address, and respond to the file read request after reading.

Specifically, utilize the file identification reading module to obtain the file identification primary key Key of the file to be read according to the received file reading request, and utilize the corresponding identification calculation module to calculate the MD5 value corresponding to the primary key Key of the file to be read according to the primary key Key . Utilize the set bit data reading module to read the first 32 MD5 values of the corresponding key value from the storage node according to the primary key Key, compare the MD5 value of the link identification and the first 32 MD5 values read in the matching module, If they are consistent, it means that the file to be read has been de-duplicated, and the file content stores the storage address of the stored file with the same content in the storage node, rather than the real content of the file itself. Remove the first 32 in the file content. For bit data, the file search module reads the storage address from the key value, locates and searches the corresponding file in the storage node according to the storage address, and responds to the file read request after reading. If not, it means that the file to be read has not been deduplicated, and the file read request is responded to after reading the file content from the key value.

Embodiment 4 of the present invention provides a deduplication device based on HDFS storage files, which can effectively remove files with duplicate content, reduce the number of files, save a lot of storage space, release memory resources, and improve system performance. At the same time, it can meet fast Storage and correct read requirements.

The device provided in the embodiment of the present invention can execute the method provided in any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

Note that the above are only preferred embodiments of the present invention and applied technical principles. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and can also include more other equivalent embodiments without departing from the concept of the present invention, and the present invention The scope is determined by the scope of the appended claims.

Claims (14)

1. A deduplication method based on HDFS storage file, is characterized in that, comprises: Compare the file fingerprint of the file to be deduplicated with the file fingerprint of the stored file; If the comparison result is the same, calculate the link identifier according to the file identifier of the file to be deduplicated; Replace the file content of the file to be deduplicated with the link ID and the storage address of the same stored file in the storage node, and store it in the storage node as the key value of the file ID of the file to be deduplicated. 2. The method according to claim 1, wherein, before comparing the file fingerprint of the file to be deduplicated with the file fingerprint of the stored file, further comprising: storing the received file in the set area in the storage node, and marking it as an area not deduplicated; Obtain files one by one from the non-deduplication processing area as files to be deduplication. 3. The method according to claim 2, wherein storing the received file into the set area in the storage node comprises: Generate a primary key for the received file as a file identifier; Converting the file content of the file into binary data, corresponding to the file identifier, and storing it in the setting area of the storage node. 4. The method according to claim 2, wherein storing the received file into the set area in the storage node comprises: According to the receiving date of the file, the received file is stored in different setting areas in the storage node. 5. The method according to claim 1, wherein calculating the link identifier according to the file identifier of the file to be deduplicated comprises: A 32-bit MD5 value is calculated for the file identifier of the file to be deduplicated as the link identifier. 6. The method according to claim 1, wherein, the file content of the file to be deduplicated is replaced with the link identifier and the storage address of the same stored file in the storage node as the file content of the file to be deduplicated. After the key value of the file identifier of the heavy file is stored in the storage node, it also includes: Rewrite the index file of the storage node according to the storage location of each file identifier and the corresponding key value in the storage node. 7. The method according to any one of claims 1-6, further comprising: Obtain the file identifier of the file to be read according to the received file read request; Calculating a corresponding link identifier according to the file identifier; Reading the set bit data of the corresponding key value from the storage node according to the file identifier; If comparing the link identifier with the set bit data, read the storage address from the key value; Locate and find the corresponding file in the storage node according to the storage address, and respond to the file read request after reading. 8. A deduplication device based on HDFS storage files, characterized in that, comprising: The fingerprint comparison module is used to compare the file fingerprint of the file to be deduplicated with the file fingerprint of the stored file; A link identification calculation module, configured to calculate a link identification according to the file identification of the file to be deduplicated if the comparison result is the same; A content replacement module, configured to replace the file content of the file to be deduplicated with the link identifier and the storage address of the same stored file in the storage node, as the key value storage of the file identifier of the file to be deduplicated to the storage node. 9. The device according to claim 8, further comprising: The file storage module is used to store the received file in the set area in the storage node before comparing the file fingerprint of the file to be deduplicated with the file fingerprint of the stored file, and mark it as not deduplicated processing area; The file acquisition module is configured to acquire files one by one from the non-deduplication processing area as files to be deduplication. 10. The device according to claim 9, wherein the file storage module comprises: A primary key generating unit, configured to generate a primary key for the received file as a file identifier; The content conversion unit is configured to convert the file content of the file into binary data, and store it in a setting area in the storage node corresponding to the file identifier. 11. The device according to claim 9, wherein the file storage module is specifically used for: According to the receiving date of the file, the received file is stored in different setting areas in the storage node. 12. The device according to claim 8, wherein the link identification calculation module is specifically used for: A 32-bit MD5 value is calculated for the file identifier of the file to be deduplicated as the link identifier. 13. The device of claim 8, further comprising: The rewriting index module is used to replace the file content of the file to be deduplicated with the link identifier and the storage address of the same stored file in the storage node as the key value of the file identifier of the file to be deduplicated After being stored in the storage node, the index file of the storage node is rewritten according to the storage location of each file identifier and the corresponding key value in the storage node. 14. The device according to any one of claims 8-13, characterized in that the device further comprises: The file identification reading module is used to obtain the file identification of the file to be read according to the received file reading request; A corresponding identification calculation module, configured to calculate a corresponding link identification according to the file identification; The set bit data reading module is used to read the set bit data of the corresponding key value from the storage node according to the file identifier; A matching module, configured to read the storage address from the key value if comparing the link identifier with the set bit data; A file search module, configured to locate and find a corresponding file in the storage node according to the storage address, and respond to the file read request after reading.