hadoop研究

包下載

創(chuàng)新互聯(lián)專注于企業(yè)網(wǎng)絡(luò)營(yíng)銷推廣、網(wǎng)站重做改版、東營(yíng)區(qū)網(wǎng)站定制設(shè)計(jì)、自適應(yīng)品牌網(wǎng)站建設(shè)、H5場(chǎng)景定制、商城網(wǎng)站制作、集團(tuán)公司官網(wǎng)建設(shè)、成都外貿(mào)網(wǎng)站建設(shè)、高端網(wǎng)站制作、響應(yīng)式網(wǎng)頁(yè)設(shè)計(jì)等建站業(yè)務(wù)，價(jià)格優(yōu)惠性價(jià)比高，為東營(yíng)區(qū)等各大城市提供網(wǎng)站開(kāi)發(fā)制作服務(wù)。

http://archive.cloudera.com/cdh5/cdh/4/

http://apache.fayea.com/hadoop/common/hadoop-2.6.4/hadoop-2.6.4.tar.gz

http://mirrors.hust.edu.cn/apache/zookeeper/zookeeper-3.4.8/zookeeper-3.4.8.tar.gz

http://apache.opencas.org/hbase/1.2.0/hbase-1.2.0-bin.tar.gz

http://download.oracle.com/otn-pub/java/jdk/8u73-b02/jdk-8u73-linux-x64.tar.gz

環(huán)境

10.200.140.58hadoop-308.99bill.com #物理機(jī) datanode zookeeper regionserver

10.200.140.59hadoop-309.99bill.com #物理機(jī) datanode zookeeper regionserver

10.200.140.60hadoop-310.99bill.com #物理機(jī) datanode zookeeper regionserver

10.200.140.45hadoop-311.99bill.com#虛擬機(jī) master

10.200.140.46hadoop-312.99bill.com#虛擬機(jī) second hmaster

修改主機(jī)名，禁用ipv6

cat /etc/profile

export JAVA_HOME=/opt/jdk1.7.0_80/

PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin

CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib

export JAVA_HOME

export PATH

export CLASSPATH

HADOOP_BASE=/opt/oracle/hadoop

HADOOP_HOME=/opt/oracle/hadoop

YARN_HOME=/opt/oracle/hadoop

PATH=$HADOOP_BASE/bin:$PATH

export HADOOP_BASE PATH

10.200.140.45 能夠免密登陸

[oracle@hadoop-311 hadoop]$ cat core-site.xml

<name>fs.defaultFS</name>

<value>hdfs://hadoop-311.99bill.com:9000</value>

</property>

<name>io.file.buffer.size</name>

</property>

</configuration>

[oracle@hadoop-311 hadoop]$ cat hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>dfs.replication</name>

</property>

<name>dfs.namenode.name.dir</name>

<value>/opt/hadoop/name</value>

</property>

<name>dfs.datanode.data.dir</name>

<value>/opt/hadoop/data/dfs</value>

</property>

<name>dfs.datanode.handler.count</name>

</property>

<name>dfs.blocksize</name>

</property>

<name>dfs.datanode.du.reserved</name>

</property>

<name>dfs.hosts.exclude</name>

<value>/opt/oracle/hadoop/etc/hadoop/slave-deny-list</value>

</property>

<name>dfs.namenode.http-address</name>

<value>hadoop-311.99bill.com:50070</value>

</property>

<name>dfs.namenode.secondary.http-address</name>

<value>hadoop-312.99bill.com:50090</value>

</property>

<name>dfs.permissions</name>

<value>false</value>

</property>

</configuration>

[oracle@hadoop-311 hadoop]$ cat mapred-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>mapreduce.framework.name</name>

</property>

<name>mapreduce.map.memory.mb</name>

</property>

<name>mapreduce.reduce.memory.mb</name>

</property>

</configuration>

定義 datanode

[oracle@hadoop-311 hadoop]$ cat slaves

hadoop-308.99bill.com

hadoop-309.99bill.com

hadoop-310.99bill.com

hadoop-env.sh

export HADOOP_LOG_DIR=$HADOOP_HOME/logs

export HADOOP_PID_DIR=/opt/oracle/hadoop

export HADOOP_SECURE_DN_PID_DIR=/opt/oracle/hadoop

export JAVA_HOME=/opt/jdk1.7.0_80/

export HADOOP_HEAPSIZE=6000

exec_time=`date +'%Y%m%d-%H%M%S'`

export HADOOP_NAMENODE_OPTS="-Xmx6g ${HADOOP_NAMENODE_OPTS}"

export HADOOP_SECONDARYNAMENODE_OPTS="-Xmx6g ${HADOOP_SECONDARYNAMENODE_OPTS}"

export HADOOP_DATANODE_OPTS="-server -Xmx6000m -Xms6000m -Xmn1000m -XX:PermSize=128M -XX:MaxPermSize=128M -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$HADOOP_LOG_DIR/gc-$(hostname)-datanode-${exec_time}.log -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=10 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=20"

[oracle@hadoop-311 hadoop]$ cat yarn-site.xml

<?xml version="1.0"?>

<name>yarn.resourcemanager.address</name>

<value>hadoop-311.99bill.com:8032</value>

</property>

<name>yarn.resourcemanager.scheduler.address</name>

<value>hadoop-311.99bill.com:8030</value>

</property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>hadoop-311.99bill.com:8031</value>

</property>

<name>yarn.resourcemanager.admin.address</name>

<value>hadoop-311.99bill.com:8033</value>

</property>

<name>yarn.resourcemanager.webapp.address</name>

<value>hadoop-311.99bill.com:8088</value>

</property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce.shuffle</value>

</property>

</configuration>

啟動(dòng)hadoop集群

第一次執(zhí)行，需要格式化namenode,以后啟動(dòng)不需要執(zhí)行此步驟。

hadoop/bin/hadoop -format

然后啟動(dòng)hadoop

hadoop/sbin/start-all.sh

啟動(dòng)完成后，如果沒(méi)有什么錯(cuò)誤，執(zhí)行jps查詢一下當(dāng)前進(jìn)程，NameNode是Hadoop Master進(jìn)程，SecondaryNameNode，ResourceManager是Hadoop進(jìn)程。

[oracle@hadoop-311 hadoop]$ jps

13332 Jps

5430 NameNode

5719 ResourceManager

三、ZooKeeper集群安裝

1. 解壓縮zookeeper-3.4.8.tar.gz并重命名zookeeper, 進(jìn)入zookeeper/conf目錄,cp zoo_sample.cfg zoo.cfg 并編輯

[oracle@hadoop-308 conf]$ cat zoo.cfg

# The number of milliseconds of each tick

tickTime=2000

maxClientCnxns=0

# The number of ticks that the initial

# synchronization phase can take

initLimit=50

# The number of ticks that can pass between

# sending a request and getting an acknowledgement

syncLimit=5

# the directory where the snapshot is stored.

# 保留快照數(shù)

autopurge.snapRetainCount=2

# Purge task interval in hours

# 清理快照時(shí)間間隔（小時(shí)）

autopurge.purgeInterval=84

dataDir=/opt/hadoop/zookeeperdata

# the port at which the clients will connect

clientPort=2181

server.1=hadoop-308:2888:3888

server.2=hadoop-309:2888:3888

server.3=hadoop-310:2888:3888

2. 新建并編輯myid文件

mkdir /opt/hadoop/zookeeperdata

echo "1" > /opt/hadoop/zookeeperdata/myid

3. 然后同步zookeeper到其他兩個(gè)節(jié)點(diǎn),然后在其他節(jié)點(diǎn)需要修改myid為相應(yīng)的數(shù)字。

啟動(dòng) zookeeper

cd /opt/oracle/zookeeper

./bin/zkServer.sh start

[oracle@hadoop-308 tools]$ jps

11939 Jps

4373 DataNode

8579 HRegionServer

四、HBase集群的安裝和配置

1. 解壓縮hbase-1.2.0-bin.tar.gz并重命名為hbase, 編輯/hbase/conf/hbase-env.sh

export HBASE_MANAGES_ZK=false

export HBASE_HEAPSIZE=4000

export JAVA_HOME=/opt/jdk1.7.0_80/

[oracle@hadoop-311 conf]$ cat hbase-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

/**

* Licensed to the Apache Software Foundation (ASF) under one

* or more contributor license agreements. See the NOTICE file

* distributed with this work for additional information

* regarding copyright ownership. The ASF licenses this file

* to you under the Apache License, Version 2.0 (the

* "License"); you may not use this file except in compliance

* with the License. You may obtain a copy of the License at

* http://www.apache.org/licenses/LICENSE-2.0

* Unless required by applicable law or agreed to in writing, software

* distributed under the License is distributed on an "AS IS" BASIS,

* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

* See the License for the specific language governing permissions and

* limitations under the License.

-->

<name>hbase.rootdir</name>

<value>hdfs://hadoop-311:9000/hbase</value>

<description>The directory shared by region servers.</description>

</property>

<name>hbase.cluster.distributed</name>

</property>

<name>hbase.master.port</name>

</property>

<name>hbase.master</name>

<value>hadoop-312</value>

</property>

<name>hbase.zookeeper.quorum</name>

<value>hadoop-308,hadoop-309,hadoop-310</value>

</property>

<name>hbase.regionserver.handler.count</name>

</property>

<name>hbase.hstore.blockingStoreFiles</name>

</property>

<name>zookeeper.session.timeout</name>

</property>

<name>hbase.regionserver.restart.on.zk.expire</name>

Zookeeper session expired will force regionserver exit.

Enable this will make the regionserver restart.

</description>

</property>

<name>hbase.replication</name>

<value>false</value>

</property>

<name>hfile.block.cache.size</name>

</property>

<name>hbase.regionserver.global.memstore.upperLimit</name>

</property>

<name>hbase.hregion.memstore.block.multiplier</name>

</property>

<name>hbase.server.thread.wakefrequency</name>

</property>

<name>hbase.master.distributed.log.splitting</name>

<value>false</value>

</property>

<name>hbase.regionserver.hlog.splitlog.writer.threads</name>

</property>

<name>hbase.client.scanner.caching</name>

</property>

<name>hbase.hregion.memstore.flush.size</name>

</property>

<name>hbase.hregion.memstore.mslab.enabled</name>

</property>

<name>hbase.coprocessor.user.region.classes</name>

<value>org.apache.hadoop.hbase.coprocessor.AggregateImplementation</value>

</property>

<name>dfs.datanode.max.xcievers</name>

<description>PRIVATE CONFIG VARIABLE</description>

</property>

</configuration>

分發(fā)hbase到其他4個(gè)節(jié)點(diǎn)

五、啟動(dòng)集群

1. 啟動(dòng)zookeeper

zookeeper/bin/zkServer.sh start

2. 啟動(dòng)Hadoop

$ hadoop/sbin/start-all.sh

修改hbase/conf/hbase-site.xml

[oracle@hadoop-311 conf]$ cat hbase-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

/**

* Licensed to the Apache Software Foundation (ASF) under one

* or more contributor license agreements. See the NOTICE file

* distributed with this work for additional information

* regarding copyright ownership. The ASF licenses this file

* to you under the Apache License, Version 2.0 (the

* "License"); you may not use this file except in compliance

* with the License. You may obtain a copy of the License at

* http://www.apache.org/licenses/LICENSE-2.0

* Unless required by applicable law or agreed to in writing, software

* distributed under the License is distributed on an "AS IS" BASIS,

* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

* See the License for the specific language governing permissions and

* limitations under the License.

-->

<name>hbase.rootdir</name>

<value>hdfs://hadoop-311:9000/hbase</value>

<description>The directory shared by region servers.</description>

</property>

<name>hbase.cluster.distributed</name>

</property>

<name>hbase.master.port</name>

</property>

<name>hbase.master</name>

<value>hadoop-312</value>

</property>

<name>hbase.zookeeper.quorum</name>

<value>hadoop-308,hadoop-309,hadoop-310</value>

</property>

<name>hbase.regionserver.handler.count</name>

</property>

<name>hbase.hstore.blockingStoreFiles</name>

</property>

<name>zookeeper.session.timeout</name>

</property>

<name>hbase.regionserver.restart.on.zk.expire</name>

Zookeeper session expired will force regionserver exit.

Enable this will make the regionserver restart.

</description>

</property>

<name>hbase.replication</name>

<value>false</value>

</property>

<name>hfile.block.cache.size</name>

</property>

<name>hbase.regionserver.global.memstore.upperLimit</name>

</property>

<name>hbase.hregion.memstore.block.multiplier</name>

</property>

<name>hbase.server.thread.wakefrequency</name>

</property>

<name>hbase.master.distributed.log.splitting</name>

<value>false</value>

</property>

<name>hbase.regionserver.hlog.splitlog.writer.threads</name>

</property>

<name>hbase.client.scanner.caching</name>

</property>

<name>hbase.hregion.memstore.flush.size</name>

</property>

<name>hbase.hregion.memstore.mslab.enabled</name>

</property>

<name>hbase.coprocessor.user.region.classes</name>

<value>org.apache.hadoop.hbase.coprocessor.AggregateImplementation</value>

</property>

<name>dfs.datanode.max.xcievers</name>

<description>PRIVATE CONFIG VARIABLE</description>

</property>

</configuration>

hbase-env.sh

export JAVA_HOME=/opt/jdk1.7.0_80/

export HBASE_CLASSPATH=/opt/oracle/hadoop/conf

export HBASE_HEAPSIZE=4000

export HBASE_OPTS="-XX:PermSize=512M -XX:MaxPermSize=512M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=10 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=20"

exec_time=`date +'%Y%m%d-%H%M%S'`

export HBASE_MASTER_OPTS="-Xmx4096m -Xms4096m -Xmn128m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$HBASE_HOME/logs/gc-$(hostname)-master-${exec_time}.log"

export HBASE_REGIONSERVER_OPTS="-Xmx8192m -Xms8192m -Xmn512m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$HBASE_HOME/logs/gc-$(hostname)-regionserver-${exec_time}.log"

export HBASE_MANAGES_ZK=fals

[oracle@hadoop-311 conf]$ cat regionservers

hadoop-308

hadoop-309

hadoop-310

分發(fā)到其他四臺(tái)

cd /opt/oracle/hbase

sh bin/start-hbase.sh

[oracle@hadoop-311 bin]$ ./hbase shell

16/03/23 20:20:47 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available

HBase Shell; enter 'help<RETURN>' for list of supported commands.

Type "exit<RETURN>" to leave the HBase Shell

Version 0.94.15-cdh5.7.1, r, Tue Nov 18 08:42:59 PST 2014

hbase(main):001:0> status

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/opt/oracle/hbase/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/opt/oracle/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

16/03/23 20:20:52 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

3 servers, 0 dead, 0.6667 average load

10.常見(jiàn)問(wèn)題

10.1.Namenode非正常關(guān)閉

在所有的hadoop環(huán)境機(jī)器上用jps命令，把所有的進(jìn)程列出，然后kill掉，再按照啟動(dòng)順序啟動(dòng)

10.2.Datanode非正常關(guān)閉

在namenode上啟動(dòng)HDFS

運(yùn)行hadoop/bin/start-all.sh

如果Datanode同時(shí)是zookeeper，還需要啟動(dòng)zookeeper

在該datanode上運(yùn)行zookeeper/bin/zkServer.sh start。

在namenode上啟動(dòng)Hbase

運(yùn)行hbase/bin/start-hbase.sh

http://10.200.140.46:60010/master-status

10.3.停止一臺(tái)非master的服務(wù)器

在該臺(tái)服務(wù)器上運(yùn)行：

hadoop/bin/hadoop-daemon.sh stop datanode

hadoop/bin/hadoop-daemon.sh stop tasktracker

hbase/bin/hbase-daemon.sh stop regionserver

在http://10.200.140.45:50070/dfshealth.jsp查看該節(jié)點(diǎn)是否已經(jīng)變成dead nodes，變成dead nodes之后，就可以停止該臺(tái)服務(wù)器

在剛停止服務(wù)的時(shí)候，看到的截圖如下：

當(dāng)停止服務(wù)成功，看到的截圖如下：

重啟服務(wù)器以后，在hadoop001上運(yùn)行，啟動(dòng)服務(wù)：

hadoop/bin/start-all.sh

hbase/bin/start-hbase.sh

11.監(jiān)控端口

11.1.Namenode監(jiān)控端口(hadoop001)：

60010,60000,50070,50030,9000,9001,10000

11.2.zookeeper監(jiān)控端口(hadoop003,hadoop004,hadoop005)

2181

11.3.Datanode監(jiān)控端口(hadoop003,hadoop004,hadoop005,hadoop006,hadoop007)

60030,50075

12、HDFS 上傳文件不均衡和Balancer太慢的問(wèn)題

Hmaster 有個(gè)start-balancer.sh

###########遷移方案

先在新機(jī)房準(zhǔn)備一套新的hadoop環(huán)境

###hadoop遷移-hbase

1 確定新hbase可以正常運(yùn)行，并且兩個(gè)集群之間的機(jī)器都可以用機(jī)器名互相訪問(wèn)到 ok

2 停掉新hbase ok

3 在兩個(gè)集群任何hadoop機(jī)器運(yùn)行下面的命令

./hadoop distcp -bandwidth 10 -m 3 hdfs://hadoop001.99bill.com:9000/hbase/if_fss_files hdfs://hadoop-312.99bill.com:9000/hbase/if_fss_files

4 使用附件的腳本，運(yùn)行

hbase org.jruby.Main ~/add_table.rb /hbase/if_fss_files

5 啟動(dòng)新hbase

###hadoop遷移-hadoop數(shù)據(jù)遷移

########整理hadoop文件,對(duì)于打包失敗的重新打包

如2014-07-24執(zhí)行

./hdfs dfs -rm -r /fss/2014-07-24

./hdfs dfs -rm -r /fss/2014-07-24.har

./hdfs dfs -mv /fss/2014-07-24a.har /fss/2014-07-24.har

##從遠(yuǎn)程fss系統(tǒng)同步到新機(jī)房本地

./hdfs dfs -copyToLocal hdfs://hadoop001.99bill.com:9000/fss/2015-04-08.har /opt/sdb/hadoop/tmp/

####從新機(jī)房本地導(dǎo)入fss系統(tǒng)

./hdfs dfs -copyFromLocal /opt/sdb/hadoop/tmp/2015-04-08.har /fss/

sleep 5

./hdfs dfs -copyFromLocal /opt/sdb/hadoop/tmp/2015-06/03-30.har /fss/2015-06

分享名稱：hadoop研究
本文地址：http://www.rwnh.cn/article16/ghcidg.html

成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供網(wǎng)站排名、網(wǎng)站設(shè)計(jì)、自適應(yīng)網(wǎng)站、ChatGPT、商城網(wǎng)站、搜索引擎優(yōu)化

聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請(qǐng)盡快告知，我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng)，如需處理請(qǐng)聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時(shí)需注明來(lái)源：創(chuàng)新互聯(lián)

猜你還喜歡下面的內(nèi)容

中文字幕日韩精品一区二区免费_精品一区二区三区国产精品无卡在_国精品无码专区一区二区三区_国产αv三级中文在线

hadoop研究