Integrating LVM volumes with Hadoop and AWS to provide elasticity.

Hadoop: Hadoop is an open source distributed processing framework that manages data processing and storage for big data applications in scalable clusters of computer servers. It’s at the center of an ecosystem of big data technologies that are primarily used to support advanced analytics initiatives, including predictive analytics, data mining and machine learning.

src: Bing Search.

Prerequisites: Read this blog first for understanding of LVM.

Integrating LVM with Hadoop

Now to use Hadoop we need a minimum of two VM’s i.e. one for Namenode and another for Datanode. So I am launching one more VM for the setting up Namenode. As my current setup is not capable of launching two VM’s at once So, I am launching One of my instance at AWS.

Now I have access to two instances, to setup Hadoop we have a package requirement for Java and Hadoop. Download Java and Hadoop from the below link to your instances. As I am using a specific version of Java and Hadoop it is hard to install it from default package manager like yum or apt.

Hadoop: https://softwareforarth.s3.ap-south-1.amazonaws.com/hadoop-1.2.1-1.x86_64.rpm

Java: https://softwareforarth.s3.ap-south-1.amazonaws.com/jdk-8u171-linux-x64.rpm

Install the above softwares with the command “rpm -i software — force

similarly install both the software in both the virtual machines. After installing both the software , It’s time to setup Namenode and Datanode.

Setting Up Namenode.

To setup Namenode edit the /etc/hadoop/hdfs-site.xml and /etc/hadoop/core-site.xml file.

Use the following code for hdfs-site.xml and core-site.xml file.

Hdfs-site.xml
core-site.xml

We here using 0.0.0.0 as a IP for NameNode because by default AWS instances doesn’t know it’s public IP addresses. Here we using /namenode folder for setting up namenode directory.

Now to start Hadoop namenode first format the namenode by using “ hadoop namenode -format” command then run “hadoop-daemon.sh start namenode” command to start the hadoop services.

Use command “hadoop dfsadmin -report” to see detailed information about hadoop cluster.

As you can see that currently no datanode is configured, So it is showing us 0 capacity.

Setting Up Datanode.

To setup datanode edit the core-site.xml and hdfs-site.xml file.

hdfs-site.xml

Here we using same folder we mounted with LVM. So we can increase or decrease the size of datanode easily.

core-site.xml

Here we are using public IP of our AWS instance on which namenode is setup. Now to start datanode we will use “hadoop-daemon.sh start datanode” command.

Start the datanode by using command “hadoop-daemon.sh start datanode”. Now our hadoop setup is complete we can check it through web UI, or by running “hadoop dfsadmin -report” command.

Hadoop detail web UI
Hadoop detail CLI

Now you guys can see currently only 12,24GB is configured. Now we have a requirement to increase the size of the datanode without shutting down the cluster. Can we achieve this??

As we configured our datanode on LVM volume, So yes we can, without stopping the contribution to cluster.

Let’s increase 3 GB to current capacity.

Now refresh the WebUI or check the report from CLI.

You can see that the size of storage increased drastically without shutting down the Cluster or Datanode.

This is How you can integrate Hadoop with LVM.

I would like to thank Vimal Sir for asking me to perform this task.

Thanks for reading this blog.

--

--

--

Hey Readers, I am tech enthusiast and an Computer Science Student. Here I am sharing various industry use cases and its solution.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Kubernetes And Need of It

https://www.mrobusinesstoday.com/iamco-extends-component-support-contract-with-afi-klm-em-for-natos-

Analyzing stock price time series with modern Fortran, Part 3

Ever since I was a little boy, I have only ever wanted to be a Doctor.

Time Complexity of an algorithm

AWS — Kinesis Overview

Multiparty Computation over Z/2^kZ

How DevOps helps your Company to Grow?

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Abhishek Sahu

Abhishek Sahu

Hey Readers, I am tech enthusiast and an Computer Science Student. Here I am sharing various industry use cases and its solution.

More from Medium

Running MediaPipe on AWS Lambda

What’s New With SAA-C03 AWS Certified Solutions Architect Associate exam?

First step with elastic security

Build your first IoT project using ESP8266 with AWS IoT Core Service |Amazon-Web-Services IoT Core…