Hadoop installation 2.7.2 Guide

Hadoop 2.7.2 installation Guide latest one by somappa Srinivasan

Step1 : Install Vmworkstion or oracle virtual Box in your machine(computer)

Download link for vmworkstation

http://filehippo.com/download_vmware-workstation-pro/

Download Oracle Virtual Box

https://www.virtualbox.org/wiki/Downloads

Step 2: Install Ubuntu os

Download Link for Ubuntu Os

https://www.ubuntu.com/download/desktop

Step 3: Update Ubuntu packages

Command : Sudo apt-get update

step 4 : Install Java 1.7 or 1.8

Command : sudo apt-get install openjdk-7-jdk

Step 5: Check whether java installed or not

Command : java -version

Step 6 : Check the Java path where JAVA installed in Ubuntu :

command : cd /usr/lib/jvm/java-1.7.0-openjdk-amd64

Step 7 : Set JAVA path in .bashrc file

Command : Sudo gedit .bashrc

Step 8 : set Java path in .bashrc file

Command :

export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64

export PATH=$PATH:$JAVA_HOME/bin

Step 9 : update the .bashrc file

Command : Source .bashrc

Step 9 : install SSH

Step 10 : Download Hadoop latest tar file hadoop 2.7.2 in apache.org website

url : https://archive.apache.org/dist/hadoop/core/hadoop-2.7.2/

SStep 11: List of out the tar file in hadoop file

Step 12: Give Permission to hadoop tar file

Step 13: Extract tar file in ubntu terminal

command : sudo tar -xvf hadoop-2.7.2.tar.gz

Step 14: List out the Hadoop Configuration files

Command :cd hadoop/etc/hadoop

Step 15 : List out bin directory files in hadoop

command : cd hadoop/sbin

Step 16 : Edit Hadoop Configuration Files

Step 17 : Create log directory in hadoop

Step 18 : Edit hadoop-env.sh file

Code :

# The java implementation to use.

export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64

Step 18 : Create mapred-site.xml file & Edit mapred-site.xml

Code :

<configuration>

<property>

<name>mapred.job.tracker</name>

<value>localhost:54311</value>

<description>The host and port that the MapReduce job tracker runs

at. If “local”, then jobs are run in-process as a single map

and reduce task.

</description>

</property>

</configuration>

Step 20 :Edit yarn-site.xml

Code :

<configuration>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

</configuration>

Step 21 : edit hdfs site.xml before editing hdfs-site.xml

Create two empty directory for namenode and datanode

Step 22 : Edit hdfs-site.xml

Code :

<configuration>

<property>

<name>dfs.replication</name>

<value>1</value>

<description>Default block replication.

The actual number of replications can be specified when the file is created.

The default is used if replication is not specified in create time.

</description>

</property>

<property>

<name>dfs.namenode.name.dir</name>

<value>file:/home/somu/namenode</value>

</property>

<property>

<name>dfs.datanode.data.dir</name>

<value>file:/home/somu/datanode</value>

</property>

</configuration>

Step 23 : Hadoop namenode format

Step 24 : Start-all .sh .. to start all deamons

command : ./start-all.sh

Step 25: to stop all deamonds

command : ./stop-all.sh

Step 26 : BROWSER UI

Namoenoe : localhost:50070

BIGDATA & HADOOP SOFTWARES

Download Link for Bigdata and Hadoop softwares list :
Download Hadoop 2.7.1 tar file

Download Hadoop 2.7.1 tar

Download Ubuntu OS

Download Ubuntu Os

Download eclipse

Download Eclipse Indigo
Download File Zilla

Download FileZilla
Cloudera 3 download Link

Download Cloudera 3

Cloudera 4

Download Clouder 4 Quick Start

Download Hadoop 1.2.0 tar

Download Hadoop 1.2.0
Download WinScp

Download Winscp Softwares
mysql Connector

Download FileZilla
Download Mysql software
https://drive.google.com/file/d/0B2-rlCGKD40NangwRGdLUXg2REE/edit

 

 

 

Hadoop 3 version Major advantages

Let’s Play with BigData more Smartly!!

Hadoop 3.x on the way!! with following features:

 Java 8 Minimum Runtime Version

Another major motivation for a new major release was bumping the minimum supported Java version to Java 8.

Intra-DataNode Balancer

Intra-DataNode balancing functionality addresses the intra-node skew that can occur when disks are added or replaced.

HDFS Erasure Coding

HDFS Erasure Coding is a major new feature, and one of the driving features for releasing Hadoop 3.0.0.

Shell Script Rewrite

The Hadoop shell scripts have been rewritten with an eye toward unifying behavior, addressing numerous long-standing bugs, improving documentation, as well as adding new functionality.

 Support for more than 2 NameNodes.

The initial implementation of HDFS NameNode high-availability provided for a single active NameNode and a single Standby NameNode. By replicating edits to a quorum of three JournalNodes, this architecture is able to tolerate the failure of any one node in the system.

However, some deployments require higher degrees of fault-tolerance. This is enabled by this new feature, which allows users to run multiple standby NameNodes. For instance, by configuring three NameNodes and five JournalNodes, the cluster is able to tolerate the failure of two nodes rather than just one.

Default ports of multiple services have been changed.
Support for Multiple Standby NameNodes

reference : https://www.quora.com/What-changes-can-be-expected-in-Hadoop-3