CN116737684A - An acceleration method for HDFS short-circuit writing - Google Patents

Abstract

The invention relates to the technical field of big data storage, and provides an acceleration method for HDFS short circuit writing, which comprises the following steps: the HDFS client, the HDFS cluster has two types of nodes and operates in a manager-worker mode, namely a NameNode (manager) and a plurality of datanodes (workers), and the acceleration method comprises the following steps: s1: creating a file and applying for Block, S2: creating Block information according to a Block allocation strategy, and S3: create RBW files of Block and Meta, S4: creating a local Java stream write Block and a Meta file, and S5: a Block information archiving record; the invention shortens the logic link for writing data and simplifies the data packaging and checking flow by stripping the logic of the data writing file and directly writing the local output stream into the disk, thereby improving the data writing efficiency.

Description

An acceleration method for HDFS short-circuit writing

Technical field

The invention relates to the technical field of big data storage, and more specifically to an acceleration method for HDFS short-circuit writing.

Background technique

HDFS, the Hadoop Distributed File System, is an open source distributed file system optimized for data appending and reading. It has the characteristics of portability, high fault tolerance and large-scale horizontal expansion. As the underlying platform for massive data, HDFS stores massive amounts of structured and unstructured data, supporting rich application scenarios such as complex queries and interactive analysis; HDFS performance issues will affect all big data systems and applications, so HDFS Optimizing storage performance is crucial.

Currently, HDFS writes data through the service provided by DataXceiverServer to establish a Socket service to accept various requests from the client. Each request has a different operation code. The server determines the request type through the operation code type; however, the above-mentioned native HDFS data writing process is relatively complicated. , and the client and server perform a large number of packet encapsulation, unpacking and data verification calculations, which further affects performance; in addition, even if the client and Datanode are on the same physical machine, the data they send needs to pass through the local loopback network (lo) transmission, causing additional machine bandwidth overhead. At present, the data writing process of HDFS is limited by unpacking verification and network pressure, which makes it impossible to truly utilize the performance of the disk. The writing speed is slow and has a great impact on the business. Therefore, the present invention provides an acceleration method for HDFS short-circuit writing.

Contents of the invention

In order to solve the above technical problems, the present invention provides an acceleration method for HDFS short-circuit writing. By stripping off part of the DataNode file writing process to the local, the logical link of writing data is shortened, the data packaging and verification process is simplified, and the network bandwidth is reduced. Consumption, improve the efficiency of writing data to solve the current problems of complex native HDFS data writing process, low performance and additional bandwidth consumption.

The specific technical solutions of the present invention are as follows:

An HDFS short-circuit write acceleration method, including: HDFS client, HDFS cluster has two types of nodes, and runs in a manager-worker mode, that is, one NameNode (manager) and multiple DataNode (workers), as described The steps of the acceleration method are as follows:

S1: Create a file and apply for a Block. The HDFS client initiates an RPC to the NameNode, creates a file and applies for the first Block;

S2: Create Block information according to the Block allocation policy, use NameNode to create Block information according to the Block allocation policy and return it to the client;

S3: Create the RBW files of Block and Meta. The client constructs the short-circuit write output stream ShortCircuitWriteOutputStream and initiates a request to the local DataNode1 to create the RBW files of Block and Meta. DataNode1 returns the descriptor information of the Block and Meta files to the client;

S4: Create a local Java stream to write Block and Meta files. The client creates a local Java stream and directly writes the Block and Meta files located on the local disk. After the client writes the Block and Meta, it submits the Block information to the local DataNode1;

S5: Block information archiving record. The client submits Block information to the NameNode. The NameNode accepts the submitted Block request and completes the Block information archiving record.

As a technical solution of the present invention, in step S1, the Block information includes BlockID, BlockLocation and other information.

As a technical solution of the present invention, in step S3, after the client obtains the Block information from the NameNode, it determines whether it contains a local node based on the Block's location information BlockLocation; if it exists, it calls the interface in ShortCircuitWriteOutputStream to implement the data writing process. ; If it does not exist, call back the DFSOutputStream interface and use the native data writing process.

As a technical solution of the present invention, in step S4, the logic of writing files by the DataNode is stripped off, and the local output stream is used to directly write to the disk, thus shortening the logical link for writing data.

Compared with the prior art, the present invention has the following beneficial effects:

1. The present invention strips the logic of DataNode writing files and instead uses the local output stream to directly write to the disk, shortening the logical link of writing data, simplifying the data packaging and verification process, thereby improving the efficiency of writing data.

2. The present invention reduces the processing pressure on the DataNode by stripping the data writing logic from the DataNode; and changes the original pipeline streaming data to a local stream to directly write to the disk, which reduces the local loopback network bandwidth pressure and improves disk IO processing. ability.

Description of drawings

Figure 1 is a flow chart of HDFS native writing according to the present invention;

Figure 2 is an overall architecture diagram of short-circuit writing according to the present invention;

Figure 3 is a specific flow chart of short-circuit writing according to the present invention.

Detailed ways

The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate the invention but are not intended to limit the scope of the invention.

As shown in Figure 2-3, an HDFS short-circuit write acceleration method includes: HDFS client. The steps of the acceleration method are as follows:

S1: Create a file and apply for a Block. The HDFS client initiates an RPC to the NameNode, creates a file and applies for the first Block;

S2: Create Block information according to the Block allocation policy, use NameNode to create Block information according to the Block allocation policy and return it to the client;

S3: Create the RBW files of Block and Meta. The client constructs the short-circuit write output stream ShortCircuitWriteOutputStream and initiates a request to the local DataNode1 to create the RBW files of Block and Meta. DataNode1 returns the descriptor information of the Block and Meta files to the client;

S4: Create a local Java stream to write Block and Meta files. The client creates a local Java stream and directly writes the Block and Meta files located on the local disk. After the client writes the Block and Meta, it submits the Block information to the local DataNode1; it writes the file by stripping the DataNode. The logic is changed to use the local output stream to write directly to the disk, which shortens the logical link of writing data, simplifies the data packaging and verification process, thereby improving the efficiency of writing data, and the writing data logic is separated from the DataNode side, reducing the load on the DataNode side. dealing with stress;

S5: Block information archiving record. The client submits Block information to the NameNode. The NameNode accepts the submitted Block request and completes the Block information archiving record. At this point, the Block is visible for external reading. At the same time, the NameNode detects that the Block copy is missing (default is 2 copies, the short-circuit write scheme only writes 1 copy to the local DataNode), and initiates a Block copy recovery request. DataNode1 copies the second copy to DataNode2, completing the dual copy writing of the Block.

This invention separates the process of writing Block and Meta files from the DataNode side, and instead uses local streams to quickly write data into files, thereby simplifying the large-scale data verification process in the native writing process, reducing the request pressure on the DataNode side, and improving the writing performance. Close to local file system performance. At the same time, the present invention uses the distribution and copy detection mechanism of HDFS to automatically complete the recovery of the second copy of the Block by HDFS to ensure the reliability of the data. Since the Block can be visible to the outside world after writing the first copy, it is better than native writing 2 The copy can only be visible to the outside world after it is written, which further speeds up file writing.

The following table shows the performance comparison between HDFS native write local and modified HDFS short-circuit write local. Based on the new short-circuit write method, the write performance of HDFS is improved. Table 1 shows the single-machine performance comparison of HDFS native writing and modified short-circuit writing under different concurrency amounts in a single concurrent write scenario of 10GB data; as shown in the following table:

Table 1

As shown in Figure 1, the native writing process for HDFS:

(1) The HDFS client initiates an RPC (remote procedure call) to the NameNode, creates a file and applies for the first Block (BlockID, BlockLocation and other information);

(2) NameNode creates Block information according to the Block allocation policy and returns it to the client;

(3) The client constructs the output stream DFSOutputStream, establishes a pipeline stream based on the location information of the Block, packages the data into a packet object, and sends it to each DataNode node in the pipeline through the pipeline stream;

(4) The client sends the packet to the first DataNode node DataNode1 through the pipeline. After receiving the packet sent by the client, DataNode1 verifies the validity of the data again and forwards the packet to the next DataNode node in the pipeline. At the same time, DataNode1 writes the data and checksum in the packet to the block and meta files in the local RBW directory respectively, and all DataNode nodes in the pipeline perform the same write operation;

(5) After a packet has been written to all DataNode nodes in the pipeline, success information is recorded in the response sent to the client, indicating that a packet is successfully written to all copies. After all data is written, the client closes the pipeline stream;

(6) The client submits the Block to the NameNode. After the NameNode changes the Block status, it means that the Block has been written and is visible for external reading.

The above-mentioned native HDFS data writing process is relatively complicated, and the client and server perform a large number of packet encapsulation, unpacking and data verification calculations, which further affects performance; in addition, even if the client and Datanode are on the same physical machine, The data it sends also needs to be transmitted through the local loopback network (lo), causing additional machine bandwidth overhead. At present, the data writing process of HDFS is limited by unpacking verification and network pressure, which makes it impossible to truly utilize the performance of the disk. The writing speed is slow and has a great impact on the business.

After the present invention obtains the Block information from the NameNode through the HDFS client, it determines whether it contains a local node based on the Block's location information BlockLocation. As shown in Figure 2, if there is a local DataNode, the HDFS client constructs a short-circuit write output stream ShortCircuitWriteOutputStream. , complete the short-circuit writing process; if there is no local DataNode, the HDFS client calls the DFSOutputStream interface to use the native writing data process.

It should be noted that the short-circuit write solution still uses the HDFS API, which can be flexibly configured to enable switches, enable in specific configuration directories, call new interfaces, etc. to use short-circuit write, and maintain the original distribution file system folder HdfsDirectory mechanism unchanged.

The embodiments of the present invention are given for the purpose of illustration and description, and are not intended to be exhaustive or to limit the invention to the disclosed form. Although the present invention has been described in detail with reference to the foregoing embodiments, it will not be useful to those skilled in the art. For those skilled in the art, they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent substitutions for some of the technical features.

Claims (4)

1. An acceleration method of HDFS short circuit writing, comprising: the HDFS client is characterized in that the acceleration method comprises the following steps:

s1: creating a file and applying for a Block, and initiating RPC to NameNode by an HDFS client, creating the file and applying for a first Block;

s2: creating Block information according to the Block allocation strategy, and using NameNode to create the Block information according to the Block allocation strategy and returning the Block information to the client;

s3: creating RBW files of Block and Meta, wherein a client constructs a short-circuit write output stream (ShortCircuitWriteOutputStream), and initiates a request to a local DataNode1, creates RBW files of Block and Meta, and the DataNode1 returns descriptor information of the Block and Meta files to the client;

s4: creating a local Java stream write Block and Meta file, wherein the client creates the local Java stream to directly write the Block and Meta file on the local disk, and submits Block information to the local DataNode1 after the client writes the Block and Meta;

s5: the method comprises the steps that a client side submits Block information to a NameNode, the NameNode receives the Block request, and the Block information archiving record is completed.

2. The acceleration method of HDFS short circuit writing of claim 1, wherein: in step S1, the Block information includes a BlockID and BlockLocation information.

3. The acceleration method of HDFS short circuit writing according to claim 2, wherein: in step S3, after the client acquires the Block information from the NameNode, determining whether the Block information includes a local node according to the Block location information; if so, calling an interface in the shortcircuit writeoutputstream to realize a data writing flow; if not, the callback DFSOutputStream interface uses the native write data flow.

4. The acceleration method of HDFS short circuit writing of claim 1, wherein: in step S4, the logic of writing the file by the DataNode is stripped, and the local output stream is used to directly write the disk, so as to shorten the logic link for writing the data.