{"id":55,"date":"2016-11-13T09:37:38","date_gmt":"2016-11-13T15:37:38","guid":{"rendered":"http:\/\/curiosity.roguepenguin.net\/?p=55"},"modified":"2019-05-13T06:51:27","modified_gmt":"2019-05-13T12:51:27","slug":"installing-hadoop-on-os-x-el-capitan-and-probably-sierra","status":"publish","type":"post","link":"http:\/\/curiosity.roguepenguin.net\/?p=55","title":{"rendered":"Installing Hadoop on OS X El Capitan (And probably Sierra)"},"content":{"rendered":"

This information is primarily sourced (read heavily copied and modified) from: getblueshift.com<\/a> and references stackoverflow.com<\/a>.  This also assumes you have homebrew<\/a> and Java 1.7+<\/a> installed.<\/p>\n

When I did this it installed Hadoop 2.7.3<\/p>\n

Steps:
\n1.<\/strong>  Set JAVA_HOME<\/code> in your bash profile.<\/p>\n

  $ export JAVA_HOME=$(\/usr\/libexec\/java_home)<\/code><\/p>\n

2.<\/strong>  Install hadoop with brew, as of this writing it will download and install 2.7.3<\/p>\n

  $ brew install hadoop<\/code><\/p>\n

3.<\/strong>  To make hadoop work on a single node cluster you have to go through several steps outlined here, here are the steps in brief<\/p>\n

4.<\/strong>  Setup ssh to connect to localhost without login<\/p>\n

  $ ssh-keygen -t dsa -P '' -f ~\/.ssh\/id_dsa<\/code>
\n  $ cat ~\/.ssh\/id_dsa.pub >> ~\/.ssh\/authorized_keys<\/code><\/p>\n

5.<\/strong> Test being able to login, if you are not able to you have to turn on Remote Login in System Preferences -> Sharing<\/p>\n

  $ ssh localhost<\/code><\/p>\n

6.<\/strong>  Brew installs Hadoop usually in \/usr\/local\/Cellar\/hadoop\/<\/p>\n

  $ cd \/usr\/local\/Cellar\/hadoop\/2.7.3<\/code><\/p>\n

Note that the version number may be different for your install<\/strong><\/p>\n

7.<\/strong>  Edit following config files in directory \/usr\/local\/Cellar\/hadoop\/2.7.3\/libexec\/etc\/hadoop<\/p>\n

  $ vi hdfs-site.xml<\/code><\/p>\n

    <configuration><\/span>\n      <property><\/span>\n        <name><\/span>dfs.replication<\/name><\/span>\n        <value><\/span>1<\/value><\/span>\n      <\/property><\/span>\n    <\/configuration>\n<\/span><\/code><\/pre>\n
  $ vi core-site.xml<\/code><\/p>\n
    <configuration><\/span>\n      <property><\/span>\n        <name><\/span>fs.defaultFS<\/name><\/span>\n        <value><\/span>hdfs:\/\/localhost:9000<\/value><\/span>\n      <\/property><\/span>\n    <\/configuration><\/span><\/code><\/pre>\n
  $ vi mapred-site.xml<\/code><\/p>\n
    <configuration><\/span>\n      <property><\/span>\n        <name><\/span>mapreduce.framework.name<\/name><\/span>\n        <value><\/span>yarn<\/value><\/span>\n      <\/property><\/span>\n    <\/configuration><\/span><\/code><\/pre>\n
  $ vi yarn-site.xml<\/code><\/p>\n
    <configuration><\/span>\n      <property><\/span>\n        <name><\/span>yarn.nodemanager.aux-services<\/name><\/span>\n        <value><\/span>mapreduce_shuffle<\/value><\/span>\n      <\/property><\/span>\n      <property><\/span>\n        <name><\/span>yarn.resourcemanager.address<\/name><\/span>\n        <value><\/span>127.0.0.1:8032<\/value><\/span>\n      <\/property><\/span>\n      <property><\/span>\n        <name><\/span>yarn.resourcemanager.scheduler.address<\/name><\/span>\n        <value><\/span>127.0.0.1:8030<\/value><\/span>\n      <\/property><\/span>\n      <property><\/span>\n        <name><\/span>yarn.resourcemanager.resource-tracker.address<\/name><\/span>\n        <value><\/span>127.0.0.1:8031<\/value><\/span>\n      <\/property><\/span>\n    <\/configuration><\/span><\/code><\/pre>\n
8.<\/strong>  Format and start HDFS and Yarn.  See Troubleshooting Note<\/strong> at the bottom<\/p>\n
  $ cd \/usr\/local\/Cellar\/hadoop\/2.7.3
\n$ .\/bin\/hdfs namenode -format
\n$ .\/sbin\/start-dfs.sh
\n$ .\/bin\/hdfs dfs -mkdir \/user
\n$ .\/bin\/hdfs dfs -mkdir \/user\/<username>
\n$ .\/sbin\/start-yarn.sh<\/code><\/p>\n
9.<\/strong>  Hadoop talks to itself using two addresses, localhost and <machine-name>.  Out of the box I had communication issues.  The three address properties we added to yarn-site.xml fix one of the communication issues.  The other will occur when MapReduce starts to run.  It will try to connect to <machine-name> (Serenity in my case) but won’t be able to find it.  To fix this issue we need to modify \/etc\/hosts<\/p>\n
  $ sudo vi \\etc\\hosts<\/code><\/p>\n
The very first line will be:
\n    127.0.0.1       localhost<\/code>
\nAnd we need to change it to:
\n    127.0.0.1       localhost Serentiy<\/code>
\nThis defines both “localhost” and “Serentiy” as aliases for 127.0.0.1<\/p>\n
10.<\/strong>  Test examples code that came with the hadoop version<\/p>\n
  $ .\/bin\/hdfs dfs -put libexec\/etc\/hadoop input
\n$ .\/bin\/hadoop jar libexec\/share\/hadoop\/mapreduce\/hadoop-mapreduce-examples-2.4.1.jar grep input output 'dfs[a-z.]+'
\n$ .\/bin\/hdfs dfs -get output output
\n$ cat output\/*<\/code><\/p>\n
11.<\/strong>  Remove tmp files<\/p>\n
  $ .\/bin\/hdfs dfs -rmr \/users\/\/input
\n$ .\/bin\/hdfs dfs -rmr \/users\/\/ouput
\n$ rm -rf output\/<\/code><\/p>\n
12.<\/strong>  Stop HDFS and Yarn after you are done<\/p>\n
  $ .\/sbin\/stop-yarn.sh
\n$ .\/sbin\/stop-dfs.sh<\/code><\/p>\n
13.<\/strong>  Add HADOOP_HOME and CONFIG to bashrc for future use<\/p>\n
  $ export HADOOP_HOME=\/usr\/local\/Cellar\/hadoop\/2.7.3
\n$ export HADOOP_CONF_DIR=$HADOOP_HOME\/libexec\/etc\/hadoop<\/code><\/p>\n
Troubleshooting Note<\/h4>\n
Note:<\/strong> HDFS attempts to put the HDFS folder in\/home<\/code>.  On one of my machines this failed due to permission issues and I had to move the HDFS to a new location.  I chose \/usr\/local\/share\/hduser\/.  If you need to move the folder location, you will need to create the directories and the add two more properties to HDFS<\/p>\n
  $ mkdir -p \/usr\/local\/share\/hduser\/mydata\/hdfs\/namenode<\/code>
\n  $ mkdir -p \/usr\/local\/share\/hduser\/mydata\/hdfs\/datanode<\/code>
\n  $ mkdir -p \/usr\/local\/share\/hduser\/tmp<\/code><\/p>\n
  $ vi hdfs-site.xml<\/code><\/p>\n
    <configuration><\/span>\n      <property><\/span>\n        <name><\/span>dfs.replication<\/name><\/span>\n        <value><\/span>1<\/value><\/span>\n      <\/property><\/span><\/code><property><\/span>\n        <name><\/span>dfs.namenode.name.dir<\/name><\/span>\n        <value>file:\/usr\/local\/share\/hduser\/mydata\/hdfs\/namenode<\/span><\/value><\/span>\n      <\/property>\n<\/span><\/code>      <property><\/span>\n        <name><\/span><\/code>dfs.datanode.name.dir<\/code><\/name><\/span> \n        <value>file:\/usr\/local\/share\/hduser\/mydata\/hdfs\/datanode<\/span><\/value>\n<\/span>      <\/property>\n<\/span><\/code>    <\/configuration>\n\n<\/span>  $ vi core-site.xml<\/code><\/pre>\n
    <configuration><\/span>\n      <property><\/span>\n        <name><\/span>fs.defaultFS<\/name><\/span>\n        <value><\/span>hdfs:\/\/localhost:9000<\/value><\/span>\n      <\/property>\n<\/span><\/code>      <property><\/span>\n        <name>hadoop.tmp.dir<\/span><\/name><\/span>\n        <value>file:\/usr\/local\/share\/hduser\/tmp<\/span><\/value><\/span>\n      <\/property>\n<\/span><\/code>    <\/configuration>\n<\/span><\/code><\/pre>\n
If you change the folder, you will need to rerun “hdfs namenode -format<\/code>” to format the new location.<\/p>\n","protected":false},"excerpt":{"rendered":"
This information is primarily sourced (read heavily copied and modified) from: getblueshift.com and references stackoverflow.com.  This also assumes you have homebrew and Java 1.7+ installed. When I did this it installed Hadoop 2.7.3 Steps: 1.  Set JAVA_HOME in your bash profile.   $ export JAVA_HOME=$(\/usr\/libexec\/java_home) 2.  Install hadoop with brew, as of this writing it will download and […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[16,6],"tags":[],"class_list":["post-55","post","type-post","status-publish","format-standard","hentry","category-hadoop","category-programming"],"_links":{"self":[{"href":"http:\/\/curiosity.roguepenguin.net\/index.php?rest_route=\/wp\/v2\/posts\/55","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/curiosity.roguepenguin.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/curiosity.roguepenguin.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/curiosity.roguepenguin.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/curiosity.roguepenguin.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=55"}],"version-history":[{"count":4,"href":"http:\/\/curiosity.roguepenguin.net\/index.php?rest_route=\/wp\/v2\/posts\/55\/revisions"}],"predecessor-version":[{"id":60,"href":"http:\/\/curiosity.roguepenguin.net\/index.php?rest_route=\/wp\/v2\/posts\/55\/revisions\/60"}],"wp:attachment":[{"href":"http:\/\/curiosity.roguepenguin.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=55"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/curiosity.roguepenguin.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=55"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/curiosity.roguepenguin.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=55"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
14.<\/strong>  Complete!  See the original article<\/a> for a note about installing Pig<\/p>\n