hadoop集群环境搭建-白红宇的个人博客

hadoop集群环境搭建

发布日期：2021-05-16 10:23:37 浏览次数：11 分类：技术文章

本文共 9154 字，大约阅读时间需要 30 分钟。

经小编多次版本测试，终于把hadoop集群搞起来了...

一.环境准备

1. hadoop2.6

2.Hbase0.98.9

3.hive0.13.1

4.zookeeper-3.4.6

5.sqoop-1.4.5

6.jre7.65(jdk7.65)

7.装备有ssh的cenos6,5的linux机器

二 .环境配置

1.ssh无密码认证（方便hadoop集群启动，否有你几个节点就要输入几次密码，当你多达N台的时候）

步骤:

1.ssh-keygen -t rsa -P '' 生成公钥/私钥对

2.cd /home/当前用户/.ssh(root 的目录为 :/root/.ssh ) 底下有两文件 id_rsa id_rsa.pub

3.cat id_rsa.pub >>authorized_keys

4. ssh localhost 成功，则单台配置成功，不行赋个权给 chmod 600 authorized_keys

5.同理用1的方法给其它机器生成公钥/私钥对

6. scp id_rsa.pub root@第一台主机IP:/root/.ssh/id_rsa72.pub

7.cat id_rsa72.pub >> authorized_keys 把所有机子公钥到一文件中

8.scp authorized_keys root@第一台主机IP:/root/.ssh/authorized_keys 把最全的公钥发到别的机器上

9.现在每台机子连接都不需要密码了

2.jdk安装(jdk是hadoo信赖的基本环境)

步骤：

1.用 tar -xzvf 解压 jdk1.7.0_65

2. vi /etc/profile

export JAVA_HOME=/home/hadoop/app/jdk1.7.0_65

export PATH=$JAVA_HOME/bin:$PATH

export CLASSPATH=.

3. source /etc/profile

4. 用javac | java -version 测试成功

3.hadoop2.6配置

步骤:

1.tar -xzvf 解压包

2. vi /etc/hosts 配置机器别名为方法hadoop配置

10.77.17.93 wzzcmaster

10.77.17.70 wzzchadoop1

10.77.17.72 wzzchadoop2

10.77.17.73 wzzchadoop3

3. 用mkdir 在hadoop要目录创建 dfs dfs/name dfs/data tmp 4个目录

4.把liv/native文件替换为64位的否则可能无法加载本地包(64位机才要替换)

5. 用vi 分别编辑

etc/hadoop/hadoop-env.sh

export JAVA_HOME=/home/hadoop/app/jdk1.7.0_65

etc/hadoop/yarn-env.sh

export JAVA_HOME=/home/hadoop/app/jdk1.7.0_65

etc/hadoop/core-site.xml


      
        
       
        fs.defaultFS
         
       
        hdfs://wzzcmaster:9000
        
       
        
       
        hadoop.tmp.dir
         
       
        file:/home/hadoop/app/hadoop-2.6.0/tmp
         
       
        Abasefor other temporary directories.

etc/hadoop/hdfs-site.xml


     
     
     
     
      
        
       
        dfs.namenode.secondary.http-address
         
       
        master:9001
        
        
         
       
        dfs.namenode.name.dir
          
       
        file:/home/hadoop/app/hadoop-2.6.0/dfs/name
        
       
        
       
        dfs.datanode.data.dir
         
       
        file:/home/hadoop/app/hadoop-2.6.0/dfs/data
         
       
        
       
        dfs.replication
         
       
        3
        
       
            
       
        dfs.permissions
             
       
        false

etc/hadoop/mapred-site.xml


     
         
       
        mapreduce.framework.name
          
       
        yarn

etc/hadoop/yarn-site.xml


     
       
             
       
        yarn.nodemanager.aux-services
               
       
        mapreduce_shuffle
         
        
              
       
        yarn.nodemanager.aux-services.mapreduce_shuffle.class
                
       
        org.apache.hadoop.mapred.ShuffleHandler
         
        
             
       
        yarn.resourcemanager.hostname
               
       
        wzzcmaster

6使用./bin/hdfs namenode -format格式化

7 用scp把配置好的hadoop 复制到别的机子上去

8. ./sbin/start-all.sh

用jps命令

主机上出现：ResourceManager, NameNode , SecondaryNameNode
子机器上出现：DataNode,NodeManager

4. zookeeper配置

1.tar -zxvf解压

2.mkdir data logs(一定要同步机器的时间,包括BIOS时间)

3.vi conf/zoo.cfg

# The number of milliseconds of each ticktickTime=2000# The number of ticks that the initial# synchronization phase can takeinitLimit=10# The number of ticks that can pass between# sending a request and getting an acknowledgementsyncLimit=5# the directory where the snapshot is stored.# do not use /tmp for storage, /tmp here is just# example sakes.dataDir=/home/hadoop/app/zookeeper-3.4.6/datadataLogDir=/home/hadoop/app/zookeeper-3.4.6/logs# the port at which the clients will connectclientPort=2181# the maximum number of client connections.# increase this if you need to handle more clients#maxClientCnxns=60## Be sure to read the maintenance section of the# administrator guide before turning on autopurge.## http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance## The number of snapshots to retain in dataDir#autopurge.snapRetainCount=3# Purge task interval in hours# Set to "0" to disable auto purge feature#autopurge.purgeInterval=1server.1=10.77.17.93:2888:3888server.2=10.77.17.70:2888:3888server.3=10.77.17.73:2888:3888

5.vi data/myid 加上你server所对应的标号(如93机器写上1 )

6. 三台机器 ./bin/zkServer.sh start

7.用jps查看到 QuorumPeerMain 则成功

5. Hbase配置

1.tar -zxvf 解压(一定要同步各台机器的时间,包括BIOS时间)

2.分别配置

conf/hbase-env.sh

export JAVA_HOME=/home/hadoop/app/jdk1.7.0_65export HADOOP_HOME=/home/hadoop/app/hadoop-2.6.0export HBASE_MANAGES_ZK=false

conf/hbase-site.xml


     
          
       
        hbase.rootdir
           
       
        hdfs://wzzcmaster:9000/hbase
         
        
          
       
        hbase.cluster.distributed
           
       
        true
         
        
          
       
        hbase.master
           
       
        hdfs://wzzcmaster:6000
         
        
          
       
        hbase.zookeeper.quorum
           
       
        wzzcmaster,wzzchadoop1,wzzchadoop3
         
          
          
       
        hbase.zookeeper.property.clientPort
           
       
        2180
         
         
          
       
        hbase.zookeeper.property.dataDir
           
       
        /home/hadoop/app/hbase-0.98.9/zookeeper

conf/regionservers

```
wzzchadoop1wzzchadoop2wzzchadoop3~
```
4.用scp命令复制到别的机子上

5.../bin/start-hbase.sh

用jps命令查看

主机 Hmaster

客机 HRegionServer

6.hive0.13.1配置

1.tar -zxvf解压

2. 编辑hive-site.xml(不包含个接mysql)


       
      
       hive.exec.scratchdir
        
      
       /home/hadoop/app/hive-0.13.1/tmp
        
      
       Scratch space for Hive jobs
      
     
       
      
       hive.exec.local.scratchdir
        
      
       /home/hadoop/app/hive-0.13.1/tmp
        
      
       Local scratch space for Hive jobs
      
     
       
      
       hive.metastore.warehouse.dir
        
      
       /home/hadoop/app/hive-0.13.1/warehouse
        
      
       location of default database for the warehouse
      
     
       
      
       hive.querylog.location
        
      
       /home/hadoop/app/hive-0.13.1/logs
        
      
           Location of Hive run time structured log file  
      
     
       
      
       hive.aux.jars.path
        
      
       file:///home/hadoop/app/hive-0.13.1/lib/hive-hbase-handler-0.13.1.jar,file:///home/hadoop/app/hbase-0.98.9/lib/protobuf-java-2.5.0.jar,file:///home/hadoop/app/hbase-0.98.9/lib/hbase-client-0.98.9-hadoop2.jar,file:///home/hadoop/app/hbase-0.98.9/lib/hbase-common-0.98.9-hadoop2.jar,file:///home/hadoop/app/hbase-0.98.9/lib/hbase-common-0.98.9-hadoop2-tests.jar,file:///home/hadoop/app/hbase-0.98.9/lib/hbase-protocol-0.98.9-hadoop2.jar,file:///home/hadoop/app/hbase-0.98.9/lib/hbase-server-0.98.9-hadoop2.jar,file:///home/hadoop/app/hbase-0.98.9/lib/htrace-core-2.04.jar,file:///home/hadoop/app/hbase-0.98.9/lib/zookeeper-3.4.6.jar,file:///home/hadoop/app/hbase-0.98.9/lib/guava-12.0.1.jar
      
     
                     
      
       hive.zookeeper.quorum
                      
      
       wzzcmaster,wzzchadoop1,wzzchadoop3
                      
      
       The list of  ZooKeeper servers to talk to. This is only needed for read/write  locks.

编辑hive-site.xm（后面加）

export JAVA_HOME=/home/hadoop/app/jdk1.7.0_65export HIVE_HOME=/home/hadoop/app/hive-0.13.1export HADOOP_HOME=/home/hadoop/app/hadoop-2.6.0

3. 可能要把 hive.aux.jars.path 包放在hive lib 下加个 hbase lib下 htrace-core-2.04.jar

4. ./bin/hive 进去

7.sqoop-1.4..5配置

1.tar -zxvf解压

2.编辑 sqoop-evn.sh

#Set path to where bin/hadoop is availableexport HADOOP_COMMON_HOME=/home/hadoop/app/hadoop-2.6.0#Set path to where hadoop-*-core.jar is availableexport HADOOP_MAPRED_HOME=/home/hadoop/app/hadoop-2.6.0#set the path to where bin/hbase is availableexport HBASE_HOME=/home/hadoop/app/hbase-0.98.9#Set the path to where bin/hive is availableexport HIVE_HOME=/home/hadoop/app/hive-0.13.1#Set the path for where zookeper config dir isexport ZOOCFGDIR=/home/hadoop/app/zookeeper-3.4.6

注释掉bin/configure-sqoop文件里

##Moved to be a runtime check in sqoop.#if[ ! -d "${HCAT_HOME}" ]; then#  echo "Warning: $HCAT_HOME does notexist! HCatalog jobs will fail."#  echo 'Please set $HCAT_HOME to the root ofyour HCatalog installation.'#fi#if[ ! -d "${ACCUMULO_HOME}" ]; then#  echo "Warning: $ACCUMULO_HOME does notexist! Accumulo imports will fail."#  echo 'Please set $ACCUMULO_HOME to the rootof your Accumulo installation.'#fi#Add HCatalog to dependency list#if[ -e "${HCAT_HOME}/bin/hcat" ]; then# TMP_SQOOP_CLASSPATH=${SQOOP_CLASSPATH}:`${HCAT_HOME}/bin/hcat-classpath`#  if [ -z "${HIVE_CONF_DIR}" ]; then#   TMP_SQOOP_CLASSPATH=${TMP_SQOOP_CLASSPATH}:${HIVE_CONF_DIR}#  fi#  SQOOP_CLASSPATH=${TMP_SQOOP_CLASSPATH}#fi#Add Accumulo to dependency list#if[ -e "$ACCUMULO_HOME/bin/accumulo" ]; then#  for jn in `$ACCUMULO_HOME/bin/accumuloclasspath | grep file:.*accumulo.*jar |cut -d':' -f2`; do#    SQOOP_CLASSPATH=$SQOOP_CLASSPATH:$jn#  done#  for jn in `$ACCUMULO_HOME/bin/accumuloclasspath | grep file:.*zookeeper.*jar |cut -d':' -f2`; do#    SQOOP_CLASSPATH=$SQOOP_CLASSPATH:$jn#  done#fi

3.使用以下命令

sqoop import --connect jdbc:oracle:thin:@10.77.10.194:1521:orcl --username sa --password sa --table RR --hbase-table rr --column-family SEQUENCEID --hbase-row-key SEQUENCEID -m 1 --hbase-create-table

注意：必须oracle IP权限表中必须要集群所有机器的IP

三.最后祝贺大家环境都一次性搞成功

转载地址：https://blog.csdn.net/y3over/article/details/48438819 如侵犯您的版权，请留言回复原文章的地址，我们会给您删除此文章，给您带来不便请您谅解！

上一篇：JAVA对象引用

下一篇：Redis分布式锁

发表评论

关于作者

喝酒易醉，品茶养心，人生如梦，品茶悟道，何以解忧？唯有杜康！

-- 愿君每日到此一游！

发表评论

最新留言

关于作者

推荐文章