Apache Hadoop 3.0 Installation on Windows is a short and practical guide for bigdata engineersto get their hands dirty. Since Hadoop 3.0 is not yet available with Cloudera CDH 6.x or Hortonworks HDP 3.x, this guide navigate you through the installation steps without cyngwin.
Since Hadoop 3.0 New Features are based on Java 1.8, you need following preparation before you start Hadoop 3.0 installation.
Java 1.8 For Hadoop 3.0 Installation on Window
You must have administrative priviledge to install JDK 1.8 on your windows machine. You can visit Oracle website and download the binaries to install it.
Once it is installed, or already installed, you can run java -version command on windows command prompt to validate the installation.
If you have trouble running this command, you can check the windows path and JAVA_HOME variable.
Download & Extract Hadoop 3.0 Binaries
Download the latest Hadoop 3.0 Installation on Windows bundlefrom its official website.General availability (GA) marks a point of quality and stability for the release series that indicates its ready for production use. You can also download the source code for this release which is around ~25Mb in size. Hadoop 3.0 binaries will be somewhere around ~250Mb in size
Since Hadoop 3.0 is built on Java, there is no separate distribution for Unix or Windows. All the binaries are byte-code which can run anywhere.
On successful download, validate the size of Hadoop 3.0 bundle.
While extracting the tar file, you may also find unzipping error as shown below. To avoid this, I extracted the tar in a UNIX machine and transferred the untar version to windows machine.
Once extracted, the folder in windows look like this
Windows Path Setup for Hadoop 3.0
Now we need to check and setup the JAVA_HOME and HADOOP_HOME path
Same way, set up the JAVA_HOME and java\bin folder in the path variables and verify those variables from command prompt
Configuration : HDFS 3.0 Installation on Windows
Edit fileC:/hadoop_3_x/hadoop-3.0.0/etc/hadoop/core-site.xml, paste below xml paragraph and save this file.
configuration property namefs.defaultFS/name valuehdfs://localhost:9000/value /property /configuration
Rename “mapred-site.xml.template” to “mapred-site.xml” and edit this file C:/hadoop_3_x/hadoop-3.0.0/etc/hadoop/mapred-site.xml, paste below xml paragraph and save this file.
configuration property namemapreduce.framework.name/name valueyarn/value /property /configuration
Edit file C:/hadoop_3_x/hadoop-3.0.0/etc/hadoop/hdfs-site.xml, paste below xml paragraph and save this file.
configuration property namedfs.replication/name value1/value /property property namedfs.namenode.name.dir/name valueC:/hadoop_3_x/hadoop-3.0.0/data/namenode/value /property property namedfs.datanode.data.dir/name valueC:/hadoop_3_x/hadoop-3.0.0/data/datanode/value /property /configuration
Edit file C:/hadoop_3_x/hadoop-3.0.0/etc/hadoop/yarn-site.xml, paste below xml paragraph and save this file.
configuration property nameyarn.nodemanager.aux-services/name valuemapreduce_shuffle/value /property property nameyarn.nodemanager.auxservices.mapreduce.shuffle.class/name valueorg.apache.hadoop.mapred.ShuffleHandler/value /property /configuration
Edit file C:/hadoop_3_x/hadoop-3.0.0/etc/hadoop/hadoop-env.cmd by closing the command line “JAVA_HOME=%JAVA_HOME%” instead of set “JAVA_HOME=C:\Java” (On C:\java this is path to file jdk.18.0)
@ren JAVA_HOME is required set JAVA_HOME=C:\Program Files\Java\jdk1.8.0_131
Open cmd and typing command“hdfs namenode format”. You will see
Hadoop 3.0 Installation Health Check
Previous version of Hadoop 2.x, web UI port is 50070 and it has been moved to 9870 in Hadoop 3.0. It can be accessed via web UI from localhost:9870
Hadoop 3.0 Downstream Compatibility
Following are the version compatibility matrix sheet indication the version of different Apache projects and their unit test status including basic functionality testing. This was done as part of Hadoop 3.0 Beta 1 release in Oct 2017.
[wpsm_comparison_table id=”1” class=””]
More on Hadoop 3.0 Related Topics
[wpsm_comparison_table id=”3” class=”hover-col1 “]