ABOUT ME

-

Today
-
Yesterday
-
Total
-
  • hadoop cluster 설치
    기타 2017. 2. 20. 09:00
    반응형
    OS : CentOS 7.2

    전체 노드 공통 설정
    필요한 소프트웨어 설치
    sudo yum install pdsh java-1.8.0-openjdk-devel.x86_64

    hadoop 다운로드
    tar zxvf hadoop-3.0.0-alpha1.tar.gz
    cd hadoop-3.0.0-alpha1

    hadoop 환경설정에 java 설정
    vi etc/hadoop/hadoop-env.sh
    export JAVA_HOME=/usr/lib/jvm/java-1.8.0

    vi etc/hadoop/core-site.xml
    <configuration>
        <property>
            <name>fs.defaultFS</name>
            <value>hdfs://<namenode uri>:9000</value>
        </property>
        <property>
            <name>io.file.buffer.size</name>
            <value>131072</value>
        </property>
    </configuration>


    Name 노드용 설정

    sudo mkdir -p /opt/hadoop
    sudo chown -R deploy:deploy /opt/hadoop

    vi etc/hadoop/hdfs-site.xml
    <configuration>
      <property>
        <name>dfs.namenode.name.dir</name>
        <value>/opt/hadoop</value>
      </property>
      <property>
        <name>dfs.hosts</name>
        <value><HADOOP_HOME>/etc/hadoop/workers</value>
      </property>
      <property>
        <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
        <value>False</value>
      </property>
      <property>
        <name>dfs.blocksize</name>
        <value>268435456</value>
      </property>
      <property>
        <name>dfs.namenode.handler.count</name>
        <value>100</value>
      </property>
    </configuration>

    vi etc/hadoop/workers
    datanode 추가


    Data 노드용 설정

    sudo mkdir -p /opt/hadoop

    vi etc/hadoop/hdfs-site.xml
    <configuration>
      <property>
        <name>dfs.datanode.data.dir</name>
        <value>/opt/hadoop</value>
      </property>
    </configuration>

    namenode와 datanode간에 ssh 접속이 되어야 한다.
    ssh 키 생성
    ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
    cat ~/.ssh/id_rsa.pub
    cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

    namenode에 생성된 키 내용 추가
    vi ~/.ssh/authorized_keys
    chmod 0600 ~/.ssh/authorized_keys



    Resource Manager & Node Manager 용 공통 설정

    vi etc/hadoop/yarn-site.xml
    필요에 따라 다음 항목들 옵션 설정
    yarn.acl.enable, yarn.admin.acl, yarn.log-aggregation-enable


    Resource Manager용 설정

    vi etc/hadoop/yarn-site.xml
    <configuration>
    <property>
      <name>yarn.resourcemanager.hostname</name>
      <value><resource manager uri></value>
    </property>
    <property>
      <name>yarn.resourcemanager.scheduler.class</name>
      <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
    </property>
    <property>
      <name>yarn.resourcemanager.nodes.include-path</name>
      <value><HADOOP_HOME>/etc/hadoop/workers</value>
    </property>
    <property>
      <name>yarn.scheduler.minimum-allocation-mb</name>
      <value>3072</value>
    </property>
    <property>
      <name>yarn.scheduler.maximum-allocation-mb</name>
      <value>3072</value>
    </property>
    </configuration>


    Node Manager 용 설정

    sudo mkdir -p /opt/yarn/data
    sudo mkdir -p /opt/yarn/log


    vi etc/hadoop/yarn-site.xml
    <configuration>
    <property>
      <name>yarn.nodemanager.resource.memory-mb</name>
      <value>3072</value>
    </property>
    <property>
      <name>yarn.nodemanager.vmem-pmem-ratio</name>
      <value>1</value>
    </property>
    <property>
      <name>yarn.nodemanager.local-dirs</name>
      <value>/opt/yarn/data</value>
    </property>
    <property>
      <name>yarn.nodemanager.log-dirs</name>
      <value>/opt/yarn/log</value>
    </property>
    <property>
      <name>yarn.resourcemanager.hostname</name>
      <value><resource manager uri></value>
    </property>
    </configuration>


    파일시스템 포맷하기
    bin/hdfs namenode -format <cluster_name>

    Namenode 시작하기
    bin/hdfs --daemon start namenode

    Datanode 시작하기
    bin/hdfs --daemon start datanode

    Yarn Resource manager 시작하기
    bin/yarn --daemon start resourcemanager

    Yarn Node manager 시작하기
    bin/yarn --daemon start nodemanager








    반응형

    '기타' 카테고리의 다른 글

    Apache Drill 살펴보기  (0) 2017.02.24
    CDH (Cloudera Hadoop) 설치  (0) 2017.02.22
    hadoop 싱글 노드 설치  (0) 2017.02.17
    storm 개요  (0) 2015.12.23
    Fleet 사용하기  (0) 2015.10.23

    댓글

Designed by Tistory.