hadoop cluster 설치

OS : CentOS 7.2

전체 노드 공통 설정

필요한 소프트웨어 설치

sudo yum install pdsh java-1.8.0-openjdk-devel.x86_64

hadoop 다운로드

wget http://mirror.apache-kr.org/hadoop/common/hadoop-3.0.0-alpha1/hadoop-3.0.0-alpha1.tar.gz tar zxvf hadoop-3.0.0-alpha1.tar.gz cd hadoop-3.0.0-alpha1

hadoop 환경설정에 java 설정

vi etc/hadoop/hadoop-env.sh export JAVA_HOME=/usr/lib/jvm/java-1.8.0

vi etc/hadoop/core-site.xml

<configuration>     <property>         <name>fs.defaultFS</name>         <value>hdfs://<namenode uri>:9000</value>     </property>     <property>         <name>io.file.buffer.size</name>         <value>131072</value>     </property> </configuration>

Name 노드용 설정

sudo mkdir -p /opt/hadoop

sudo chown -R deploy:deploy /opt/hadoop

vi etc/hadoop/hdfs-site.xml

<configuration>   <property>     <name>dfs.namenode.name.dir</name>     <value>/opt/hadoop</value>   </property>   <property>     <name>dfs.hosts</name>     <value><HADOOP_HOME>/etc/hadoop/workers</value>   </property>   <property>     <name>dfs.namenode.datanode.registration.ip-hostname-check</name>     <value>False</value>   </property>   <property>     <name>dfs.blocksize</name>     <value>268435456</value>   </property>   <property>     <name>dfs.namenode.handler.count</name>     <value>100</value>   </property> </configuration>

vi etc/hadoop/workers

datanode 추가

Data 노드용 설정

sudo mkdir -p /opt/hadoop

vi etc/hadoop/hdfs-site.xml

<configuration>   <property>     <name>dfs.datanode.data.dir</name>     <value>/opt/hadoop</value>   </property> </configuration>

namenode와 datanode간에 ssh 접속이 되어야 한다.

ssh 키 생성

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

cat ~/.ssh/id_rsa.pub

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

namenode에 생성된 키 내용 추가

vi ~/.ssh/authorized_keys

chmod 0600 ~/.ssh/authorized_keys

Resource Manager & Node Manager 용 공통 설정

vi etc/hadoop/yarn-site.xml

필요에 따라 다음 항목들 옵션 설정

yarn.acl.enable, yarn.admin.acl, yarn.log-aggregation-enable

Resource Manager용 설정

vi etc/hadoop/yarn-site.xml

<configuration> <property>   <name>yarn.resourcemanager.hostname</name>   <value><resource manager uri></value> </property> <property>   <name>yarn.resourcemanager.scheduler.class</name>   <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value> </property> <property>   <name>yarn.resourcemanager.nodes.include-path</name>   <value><HADOOP_HOME>/etc/hadoop/workers</value> </property> <property>   <name>yarn.scheduler.minimum-allocation-mb</name>   <value>3072</value> </property> <property>   <name>yarn.scheduler.maximum-allocation-mb</name>   <value>3072</value> </property> </configuration>

Node Manager 용 설정

sudo mkdir -p /opt/yarn/data

sudo mkdir -p /opt/yarn/log

vi etc/hadoop/yarn-site.xml

<configuration> <property>   <name>yarn.nodemanager.resource.memory-mb</name>   <value>3072</value> </property> <property>   <name>yarn.nodemanager.vmem-pmem-ratio</name>   <value>1</value> </property> <property>   <name>yarn.nodemanager.local-dirs</name>   <value>/opt/yarn/data</value> </property> <property>   <name>yarn.nodemanager.log-dirs</name>   <value>/opt/yarn/log</value> </property> <property>   <name>yarn.resourcemanager.hostname</name>   <value><resource manager uri></value> </property> </configuration>

파일시스템 포맷하기

bin/hdfs namenode -format <cluster_name>

Namenode 시작하기

bin/hdfs --daemon start namenode

Datanode 시작하기

bin/hdfs --daemon start datanode

Yarn Resource manager 시작하기

bin/yarn --daemon start resourcemanager

Yarn Node manager 시작하기

bin/yarn --daemon start nodemanager

참고 : http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html