CN106302662A - Hbase-based MR operation method capable of saving network flow - Google Patents

Abstract

The invention discloses an Hbase-based MR operation method capable of saving network flow, which solves the problems of high network overhead and risk of network paralysis in entitlement collection, and adopts the technical scheme that: the method comprises the following steps: (1) the InputFormat method of Mapreduce is realized; (2) acquiring all large data block information of a certain Hbase table; (3) acquiring the bottom files of the data blocks according to the data blocks; (4) taking the bottom layer files of all the obtained data blocks as input of Mapreduce; executing mapreduce by taking each bottom file as a calculation unit; (5) and executing reduce to end mapreduce.

Description

A MR operation method based on Hbase to save network traffic

technical field

The present invention relates to an MR running method based on Hbase and saving network traffic in particular.

Background technique

In today's world, companies' daily operations often generate terabytes of data. Data sources include any type of data that can be captured by internet-connected devices, websites, social media, transactional business data, and data created in other business environments. Given the volume of data being generated, real-time processing is a top challenge for many organizations.

MR is the abbreviation of mapreduce, and MapReduce is a programming model for parallel computing of large-scale data sets (greater than 1TB). The concepts "Map" and "Reduce", and their main ideas, are borrowed from functional programming languages, with features borrowed from vector programming languages. It greatly facilitates programmers to run their programs on distributed systems without distributed parallel programming. The current software implementation is to specify a Map (mapping) function to map a set of key-value pairs into a new set of key-value pairs, and specify a concurrent Reduce (reduction) function to ensure that all mapped key-value pairs are Each of the shares the same set of keys.

Use hbase to run MR, because the underlying data files of HBase itself are not all distributed on the MR running nodes. Therefore, when executing MR, the MR execution node will read data files on other nodes across the network, resulting in a lot of additional network overhead. When the cluster data reaches terabytes or petabytes, the traditional mapreduce of Hbase can easily cause a lot of network overhead, and centralization has the risk of network paralysis.

Contents of the invention

The technical task of the present invention is to provide an Hbase-based MR operation method that saves network traffic to solve the problems of large network overhead and risk of network paralysis due to centralization.

Technical task of the present invention is realized in the following manner,

An Hbase-based MR operation method for saving network traffic, the steps are as follows:

(1) Implement the InputFormat method of Mapreduce;

(2) Obtain all large data block (Region) information of a table in Hbase;

(3) Obtain their underlying files (Hfile) according to each data block;

(4) Use the obtained underlying files of all data blocks as the input of Mapreduce; use each underlying file as a calculation unit to execute mapreduce;

(5) Execute reduce and end mapreduce.

In step (4), mapreduce is executed. MapReduce achieves reliability by distributing large-scale operations on data sets to each computing unit on the network; each computing unit periodically returns its completed work and the latest status.

If a computing unit remains silent for more than a preset time interval, the main computing unit (similar to the main server in GoogleFile System) records the status of this computing unit as dead, and sends the data assigned to this computing unit to other computing units unit.

An Hbase-based MR operation method for saving network traffic of the present invention has the following advantages: combining the execution characteristics of MR and the storage characteristics of HBase data, MR is directly executed on each data file, which fundamentally solves the problem of the initial delay of Mapreduce operation. The problem of node fetching data, thus saving the network overhead very well, has a very good promotion and use value.

Description of drawings

The present invention will be further described below in conjunction with the accompanying drawings.

Accompanying drawing 1 is a flow chart of an Hbase-based MR operation method for saving network traffic.

detailed description

An Hbase-based MR operation method for saving network traffic of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

Example:

A kind of MR running method based on Hbase of the present invention saves network flow, and the steps are as follows:

(1) Implement the InputFormat method of Mapreduce;

(2) Obtain all large data block (Region) information of a table in Hbase;

(3) Obtain their underlying files (Hfile) according to each data block;

(4) Use the obtained underlying files of all data blocks as the input of Mapreduce; use each underlying file as a calculation unit to execute mapreduce;

(5) Execute reduce and end mapreduce.

In step (4), mapreduce is executed. MapReduce achieves reliability by distributing large-scale operations on data sets to each computing unit on the network; each computing unit periodically returns its completed work and the latest status.

If a computing unit remains silent for more than a preset time interval, the main computing unit (similar to the main server in GoogleFile System) records the status of this computing unit as dead, and sends the data assigned to this computing unit to other computing units unit.

1. Mapping and Simplification

In simple terms, a mapping function is to perform a specified operation on each element of a conceptual list of independent elements (for example, a list of test scores) (for example, in the previous example, it was found that all student scores were Overestimated by one point, it can define a "minus one" mapping function to fix this error.). In fact, each element is manipulated independently, and the original list is not changed, because a new list is created here to hold the new answer. That is to say, the Map operation can be highly parallelized, which is very useful for applications with high performance requirements and the requirements in the field of parallel computing.

The simplification operation refers to the proper merging of the elements of a list (continue to look at the previous example, if someone wants to know the average score of the class, how to do it? It can define a simplification function, by making the elements in the list follow the The list is halved by adding its own adjacent elements, so recursive operation until there is only one element left in the list, and then divide this element by the number of people to get the average score.). Although it is not as parallel as the mapping function, the reduction function is also useful in a highly parallel environment because the reduction always has a simple answer and the large-scale operations are relatively independent.

2. Reliable distribution

MapReduce achieves reliability by distributing large-scale operations on data sets to each node on the network; each node will periodically return its completed work and the latest status. If a node remains silent for more than a preset time interval, the master node (similar to the master server in Google File System) records the node status as dead, and sends the data assigned to this node to other nodes. Each operation uses atomic operations on named files to ensure that no conflicts between parallel threads occur; when files are renamed, the system may copy them to a name other than the task name. (to avoid side effects).

The reduction operation works in a similar way, but because the parallelism of the reduction operation is relatively poor, the master node will try to allocate the reduction operation to only one node, or the node as close as possible to the data to be operated; This feature can meet the needs of Google, because they have enough bandwidth, and their internal network does not have so many machines.

Through the above specific implementation manners, those skilled in the technical field can easily realize the present invention. However, it should be understood that the present invention is not limited to the specific embodiments described above. On the basis of the disclosed embodiments, those skilled in the art can arbitrarily combine different technical features, so as to realize different technical solutions.

Except for the technical features described in the instructions, all are known technologies by those skilled in the art.

Claims (3)

1. an MR operation method based on Hbase to save network traffic, is characterized in that the steps are as follows: (1) Implement the InputFormat method of Mapreduce; (2) Obtain all the large data block information of a table in Hbase; (3) Obtain their underlying files according to each data block; (4) Use the obtained underlying files of all data blocks as the input of Mapreduce; use each underlying file as a calculation unit to execute mapreduce; (5) Execute reduce and end mapreduce. 2. A Hbase-based MR operation method for saving network traffic according to claim 1, characterized in that in step (4), mapreduce is executed, and MapReduce distributes large-scale operations on data sets to each network on the network Each computing unit achieves reliability; each computing unit periodically returns the work it has done and the latest status. 3. A Hbase-based MR operation method for saving network traffic according to claim 2, wherein if a computing unit remains silent for more than a preset time interval, the main computing unit records that the computing unit state is dead , and send the data assigned to this computing unit to other computing units.