Hadoop集群搭建(二) HDFS
HDFS只是Hadoop最基本的一个服务,很多其他服务,都是基于HDFS展开的。所以部署一个HDFS集群,是很核心的一个动作,也是大数据平台的开始。
安装Hadoop集群,首先需要有Zookeeper才可以完成安装。如果没有Zookeeper,请先部署一套Zookeeper。另外,JDK以及物理主机的一些设置等。都请参考下文:
Hadoop集群搭建(一) Zookeeper
下面开始HDFS的安装
HDFS主机分配
192.168.67.101 c6701 --Namenode+datanode 192.168.67.102 c6702 --datanode 192.168.67.103 c6703 --datanode
1. 安装HDFS,解压hadoop-2.6.0-EDH-0u2.tar.gz
我同时下载2.6和2.7版本的软件,先安装2.6,然后在执行2.6到2.7的升级步骤
useradd hdfs echo "hdfs:hdfs" | chpasswd su - hdfs cd /tmp/software tar -zxvf hadoop-2.6.0-EDH-0u2.tar.gz -C /home/hdfs/ mkdir -p /data/hadoop/temp mkdir -p /data/hadoop/journal mkdir -p /data/hadoop/hdfs/name mkdir -p /data/hadoop/hdfs/data chown -R hdfs:hdfs /data/hadoop chown -R hdfs:hdfs /data/hadoop/temp chown -R hdfs:hdfs /data/hadoop/journal chown -R hdfs:hdfs /data/hadoop/hdfs/name chown -R hdfs:hdfs /data/hadoop/hdfs/data $ pwd /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop
2. 修改core-site.xml对应的参数
$ cat core-site.xml configuration !-- 指定hdfs的nameservice为ns -- property name fs.defaultFS /name value hdfs://ns /value /property !--指定hadoop数据临时存放目录-- property name hadoop.tmp.dir /name value /data/hadoop/temp /value /property property name io.file.buffer.size /name value 4096 /value /property !--指定zookeeper地址-- property name ha.zookeeper.quorum /name value c6701:2181,c6702:2181,c6703:2181 /value /property /configuration
3. 修改hdfs-site.xml对应的参数
cat hdfs-site.xml configuration !--指定hdfs的nameservice为ns,需要和core-site.xml中的保持一致,并且ns如果改,整个文件中,全部的ns要都修改,保持统一 -- property name dfs.nameservices /name value ns /value /property !-- ns下面有两个NameNode,分别是nn1,nn2 -- property name dfs.ha.namenodes.ns /name value nn1,nn2 /value /property !-- nn1的RPC通信地址 -- property name dfs.namenode.rpc-address.ns.nn1 /name value c6701:9000 /value /property !-- nn1的http通信地址 -- property name dfs.namenode.http-address.ns.nn1 /name value c6701:50070 /value /property !-- nn2的RPC通信地址 -- property name dfs.namenode.rpc-address.ns.nn2 /name value c6702:9000 /value /property !-- nn2的http通信地址 -- property name dfs.namenode.http-address.ns.nn2 /name value c6702:50070 /value /property !-- 指定NameNode的元数据在JournalNode上的存放位置 -- property name dfs.namenode.shared.edits.dir /name value qjournal://c6701:8485;c6702:8485;c6703:8485/ns /value /property !-- 指定JournalNode在本地磁盘存放数据的位置 -- property name dfs.journalnode.edits.dir /name value /data/hadoop/journal /value /property !-- 开启NameNode故障时自动切换 -- property name dfs.ha.automatic-failover.enabled /name value true /value /property !-- 配置失败自动切换实现方式 -- property name dfs.client.failover.proxy.provider.ns /name value org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider /value /property !-- 配置隔离机制 -- property name dfs.ha.fencing.methods /name value sshfence /value /property !-- 使用隔离机制时需要ssh免登陆 -- property name dfs.ha.fencing.ssh.private-key-files /name value /home/hdfs/.ssh/id_rsa /value /property property name dfs.namenode.name.dir /name value /data/hadoop/hdfs/name /value /property property name dfs.datanode.data.dir /name value /data/hadoop/hdfs/data /value /property property name dfs.replication /name value 2 /value /property !-- 在NN和DN上开启WebHDFS (REST API)功能,不是必须 -- property name dfs.webhdfs.enabled /name value true /value /property /configuration
4. 添加slaves文件
$ more slaves c6701 c6702 c6703
安装C6702的hdfs
5. 创建c6702的用户,并为hdfs用户ssh免密
ssh c6702 "useradd hdfs" ssh c6702 "echo "hdfs:hdfs" | chpasswd" ssh-copy-id hdfs@c6702
6. 拷贝软件
scp -r /tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz root@c6702:/tmp/software/. ssh c6702 "chmod 777 /tmp/software/*"
7. 创建目录,解压软件
ssh hdfs@c6702 "mkdir hdfs" ssh hdfs@c6702 "tar -zxvf /tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz -C /home/hdfs" ssh hdfs@c6702 "ls -al hdfs" ssh hdfs@c6702 "ls -al hdfs/hadoop*"
复制配置文件
ssh hdfs@c6702 "rm -rf /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml" ssh hdfs@c6702 "rm -rf /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml" scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml hdfs@c6702:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml hdfs@c6702:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves hdfs@c6702:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves
创建hdfs需要的目录
ssh root@c6702 "mkdir -p /data/hadoop" ssh root@c6702 " chown -R hdfs:hdfs /data/hadoop" ssh hdfs@c6702 "mkdir -p /data/hadoop/temp" ssh hdfs@c6702 "mkdir -p /data/hadoop/journal" ssh hdfs@c6702 "mkdir -p /data/hadoop/hdfs/name" ssh hdfs@c6702 "mkdir -p /data/hadoop/hdfs/data"
安装C6703的hdfs
8. 创建c6703的用户,并为hdfs用户ssh免密
ssh c6703 "useradd hdfs" ssh c6703 "echo "hdfs:hdfs" | chpasswd" ssh-copy-id hdfs@c6703
9. 拷贝软件
scp -r /tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz root@c6703:/tmp/software/. ssh c6703 "chmod 777 /tmp/software/*" 10. 创建目录,解压软件 ssh hdfs@c6703 "mkdir hdfs" ssh hdfs@c6703 "tar -zxvf /tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz -C /home/hdfs" ssh hdfs@c6703 "ls -al hdfs" ssh hdfs@c6703 "ls -al hdfs/hadoop*"
复制配置文件
ssh hdfs@c6703 "rm -rf /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml" ssh hdfs@c6703 "rm -rf /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml" scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml hdfs@c6703:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml hdfs@c6703:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves hdfs@c6703:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves
创建hdfs需要的目录
ssh root@c6703 "mkdir -p /data/hadoop" ssh root@c6703 " chown -R hdfs:hdfs /data/hadoop" ssh hdfs@c6703 "mkdir -p /data/hadoop/temp" ssh hdfs@c6703 "mkdir -p /data/hadoop/journal" ssh hdfs@c6703 "mkdir -p /data/hadoop/hdfs/name" ssh hdfs@c6703 "mkdir -p /data/hadoop/hdfs/data"
11. 启动HDFS,先启动三个节点的journalnode
/home/hdfs/hadoop-2.6.0-EDH-0u2/sbin/hadoop-daemon.sh start journalnode
检查状态
$ jps 3958 Jps 3868 JournalNode
12. 然后启动namenode,首次启动namenode之前,先在其中一个节点(主节点)format namenode信息,信息会存在于dfs.namenode.name.dir指定的路径中
name dfs.namenode.name.dir /name value /data/hadoop/hdfs/name /value
$ ./hdfs namenode -format 17/09/26 07:52:17 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = c6701.python279.org/192.168.67.101 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 2.6.0-EDH-0u2 STARTUP_MSG: classpath = /home/hdfs/hadoop-2.6.0-EDHxxxxxxxxxx STARTUP_MSG: build = http://gitlab-xxxxx STARTUP_MSG: java = 1.8.0_144 ************************************************************/ 17/09/26 07:52:17 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 17/09/26 07:52:17 INFO namenode.NameNode: createNameNode [-format] 17/09/26 07:52:18 WARN common.Util: Path /data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration. 17/09/26 07:52:18 WARN common.Util: Path /data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration. Formatting using clusterid: CID-b2f01411-862f-44b2-a6dc-7d17bd48d522 17/09/26 07:52:18 INFO namenode.FSNamesystem: No KeyProvider found. 17/09/26 07:52:18 INFO namenode.FSNamesystem: fsLock is fair:true 17/09/26 07:52:18 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000 17/09/26 07:52:18 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true 17/09/26 07:52:18 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000 17/09/26 07:52:18 INFO blockmanagement.BlockManager: The block deletion will start around 2017 Sep 26 07:52:18 17/09/26 07:52:18 INFO util.GSet: Computing capacity for map BlocksMap 17/09/26 07:52:18 INFO util.GSet: VM type = 64-bit 17/09/26 07:52:18 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB 17/09/26 07:52:18 INFO util.GSet: capacity = 2^21 = 2097152 entries 17/09/26 07:52:18 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false 17/09/26 07:52:18 INFO blockmanagement.BlockManager: defaultReplication = 2 17/09/26 07:52:18 INFO blockmanagement.BlockManager: maxReplication = 512 17/09/26 07:52:18 INFO blockmanagement.BlockManager: minReplication = 1 17/09/26 07:52:18 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 17/09/26 07:52:18 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false 17/09/26 07:52:18 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000 17/09/26 07:52:18 INFO blockmanagement.BlockManager: encryptDataTransfer = false 17/09/26 07:52:18 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000 17/09/26 07:52:18 INFO namenode.FSNamesystem: fsOwner = hdfs (auth:SIMPLE) 17/09/26 07:52:18 INFO namenode.FSNamesystem: supergroup = supergroup 17/09/26 07:52:18 INFO namenode.FSNamesystem: isPermissionEnabled = true 17/09/26 07:52:18 INFO namenode.FSNamesystem: Determined nameservice ID: ns 17/09/26 07:52:18 INFO namenode.FSNamesystem: HA Enabled: true 17/09/26 07:52:18 INFO namenode.FSNamesystem: Append Enabled: true 17/09/26 07:52:18 INFO util.GSet: Computing capacity for map INodeMap 17/09/26 07:52:18 INFO util.GSet: VM type = 64-bit 17/09/26 07:52:18 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB 17/09/26 07:52:18 INFO util.GSet: capacity = 2^20 = 1048576 entries 17/09/26 07:52:18 INFO namenode.NameNode: Caching file names occuring more than 10 times 17/09/26 07:52:18 INFO util.GSet: Computing capacity for map cachedBlocks 17/09/26 07:52:18 INFO util.GSet: VM type = 64-bit 17/09/26 07:52:18 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB 17/09/26 07:52:18 INFO util.GSet: capacity = 2^18 = 262144 entries 17/09/26 07:52:18 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 17/09/26 07:52:18 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0 17/09/26 07:52:18 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000 17/09/26 07:52:18 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 17/09/26 07:52:18 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis 17/09/26 07:52:18 INFO util.GSet: Computing capacity for map NameNodeRetryCache 17/09/26 07:52:18 INFO util.GSet: VM type = 64-bit 17/09/26 07:52:18 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB 17/09/26 07:52:18 INFO util.GSet: capacity = 2^15 = 32768 entries 17/09/26 07:52:18 INFO namenode.NNConf: ACLs enabled? false 17/09/26 07:52:18 INFO namenode.NNConf: XAttrs enabled? true 17/09/26 07:52:18 INFO namenode.NNConf: Maximum size of an xattr: 16384 17/09/26 07:52:19 INFO namenode.FSImage: Allocated new BlockPoolId: BP-144216011-192.168.67.101-1506412339757 17/09/26 07:52:19 INFO common.Storage: Storage directory /data/hadoop/hdfs/name has been successfully formatted. 17/09/26 07:52:20 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid = 0 17/09/26 07:52:20 INFO util.ExitUtil: Exiting with status 0 17/09/26 07:52:20 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at c6701.python279.org/192.168.67.101 ************************************************************/
13. standby namenode需要先执行bootstrapstandby,输出如下
[hdfs@c6702 sbin]$ ../bin/hdfs namenode -bootstrapstandby 17/09/26 09:44:58 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = c6702.python279.org/192.168.67.102 STARTUP_MSG: args = [-bootstrapstandby] STARTUP_MSG: version = 2.6.0-EDH-0u2 STARTUP_MSG: classpath = /home/hdfs/haxxx STARTUP_MSG: build = http://gitlab-xxxx STARTUP_MSG: java = 1.8.0_144 ************************************************************/ 17/09/26 09:44:58 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 17/09/26 09:44:58 INFO namenode.NameNode: createNameNode [-bootstrapstandby] 17/09/26 09:44:59 WARN common.Util: Path /data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration. 17/09/26 09:44:59 WARN common.Util: Path /data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration. ===================================================== About to bootstrap Standby ID nn2 from: Nameservice ID: ns Other Namenode ID: nn1 Other NNs HTTP address: http://c6701:50070 Other NNs IPC address: c6701/192.168.67.101:9000 Namespace ID: 793662207 Block pool ID: BP-144216011-192.168.67.101-1506412339757 Cluster ID: CID-b2f01411-862f-44b2-a6dc-7d17bd48d522 Layout version: -60 ===================================================== Re-format filesystem in Storage Directory /data/hadoop/hdfs/name ? (Y or N) y 17/09/26 09:45:16 INFO common.Storage: Storage directory /data/hadoop/hdfs/name has been successfully formatted. 17/09/26 09:45:16 WARN common.Util: Path /data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration. 17/09/26 09:45:16 WARN common.Util: Path /data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration. 17/09/26 09:45:17 INFO namenode.TransferFsImage: Opening connection to http://c6701:50070/imagetransfer?getimage=1 txid=0 storageInfo=-60:793662207:0:CID-b2f01411-862f-44b2-a6dc-7d17bd48d522 17/09/26 09:45:17 INFO namenode.TransferFsImage: Image Transfer timeout configured to 60000 milliseconds 17/09/26 09:45:17 INFO namenode.TransferFsImage: Transfer took 0.01s at 0.00 KB/s 17/09/26 09:45:17 INFO namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000000 size 351 bytes. 17/09/26 09:45:17 INFO util.ExitUtil: Exiting with status 0 17/09/26 09:45:17 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at c6702.python279.org/192.168.67.102 ************************************************************/
14. 检查状态,namenode还没有启动
[hdfs@c6702 sbin]$ jps 4539 Jps 3868 JournalNode
15. 启动standby namenode,命令和master启动的方式相同
[hdfs@c6702 sbin]$ ./hadoop-daemon.sh start namenode starting namenode, logging to /home/hdfs/hadoop-2.6.0-EDH-0u2/logs/hadoop-hdfs-namenode-c6702.python279.org.out
16. 再次检查,namenode已经启动
[hdfs@c6702 sbin]$ jps 4640 Jps 4570 NameNode 3868 JournalNode
17. 格式化zkfc,让在zookeeper中生成ha节点,在master上执行如下命令,完成格式化
[hdfs@c6701 bin]$ ./hdfs zkfc -formatZK 17/09/26 09:59:20 INFO tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at c6701/192.168.67.101:9000 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:host.name=c6701.python279.org 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_144 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/local/jdk1.8.0_144/jre 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/home/hdfs/hadoop-2.6.0-EDH-0u2/exxxx 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/home/hdfs/hadoop-2.6.0-EDH-0u2/lib/native 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.compiler= NA 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-573.el6.x86_64 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:user.name=hdfs 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hdfs 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hdfs/hadoop-2.6.0-EDH-0u2/bin 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=c6701:2181,c6702:2181,c6703:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@20deea7f 17/09/26 09:59:20 INFO zookeeper.ClientCnxn: Opening socket connection to server c6703.python279.org/192.168.67.103:2181. Will not attempt to authenticate using SASL (unknown error) 17/09/26 09:59:20 INFO zookeeper.ClientCnxn: Socket connection established to c6703.python279.org/192.168.67.103:2181, initiating session 17/09/26 09:59:20 INFO zookeeper.ClientCnxn: Session establishment complete on server c6703.python279.org/192.168.67.103:2181, sessionid = 0x35ebc5163710000, negotiated timeout = 5000 17/09/26 09:59:20 INFO ha.ActiveStandbyElector: Session connected. 17/09/26 09:59:20 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/ns in ZK. 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Session: 0x35ebc5163710000 closed 17/09/26 09:59:20 INFO zookeeper.ClientCnxn: EventThread shut down
18. 格式化完成的检查
格式成功后,查看zookeeper中可以看到 命令没确认
[zk: localhost:2181(CONNECTED) 1] ls /hadoop-ha
19. 启动zkfc,这个就是为namenode使用的
./hadoop-daemon.sh start zkfc starting zkfc, logging to /home/hdfs/hadoop-2.6.0-EDH-0u2/logs/hadoop-hdfs-zkfc-c6701.python279.org.out $ jps 4272 DataNode 4402 JournalNode 6339 Jps 6277 DFSZKFailoverController 4952 NameNode
20. 另一个节点启动zkfc,
ssh hdfs@c6702 /home/hdfs/hadoop-2.6.0-EDH-0u2/sbin/hadoop-daemon.sh start zkfc $ jps 4981 Jps 4935 DFSZKFailoverController 4570 NameNode 3868 JournalNode
21. 注意:进行初始化的时候,必须保证zk集群已经启动了。
1、在ZK中创建znode来存储automatic Failover的数据,任选一个NN执行完成即可:
sh bin/hdfs zkfc -formatZK
2、启动zkfs,在所有的NN节点中执行以下命令:
sh sbin/hadoop-daemon.sh start zkfc
22. 启动datanode
最后启动集群
/home/hdfs/hadoop-2.6.0-EDH-0u2/sbin/hadoop-daemon.sh start zkfc sh sbin/start-dfs.sh
HDFS安装过程中的重点,最后在软件启动过程中,一些初始化操作,很重要。
1. 启动全部的journalnode
2. 在namenode1上执行, hdfs namenode -formatZK
3. 在namenode1上执行, 启动namenode1,命令hadoop-daemon.sh start namenode
4. 在namenode2上执行, hdfs namenode -bootstrapstandby
5. 在namenode1上执行,格式化zkfc,在zookeeper中生成HA节点, hdfs zkfc -formatZK
6. 启动zkfc,hadoop-daemon.sh start zkfc。 有namenode运行的节点,都要启动ZKFC
7. 启动 datanode
HDFS只是Hadoop最基本的一个模块,这里已经安装完成,可以为后面的Hbase提供服务了。
原创文章,作者:ItWorker,如若转载,请注明出处:https://blog.ytso.com/54744.html
apachejavalinuxpython相关文章
- Hadoop面试
- Hadoop资源调度器
- eureka集群配置_hadoop高可用集群搭建
- Hadoop生态圈各种组件介绍
- Hadoop Spark太重,esProc SPL很轻
- 搭建hadoop集群的三种方式_hadoop集群部署
- Hadoop伪分布式搭建
- Hadoop实战_hadoop 项目实战
- hadoop生态圈相关技术_hadoop的生态
- 大数据面试题(二):Hadoop的联邦机制核心高频面试题
- hadoop集群安装
- 《使用IBCS虚拟专线搭建Hadoop集群:详细步骤与优势》
- Hadoop分布式集群搭建hadoop2.6+Ubuntu16.04详解大数据
- Hadoop(三)手把手教你搭建Hadoop全分布式集群详解大数据
- Hadoop(十七)之MapReduce作业配置与Mapper和Reducer类详解大数据
- Hadoop综合练习第七节–Hive部署测试(含MySql部署)详解大数据
- Hadoop入门进阶课程11–Sqoop介绍、安装与操作详解大数据
- Linux 下 Hadoop 2.6.0 集群环境的搭建详解大数据
- ZooKeeper学习之路 (十)Hadoop的HA集群的机架感知详解大数据
- Hadoop体系结构详解大数据
- Hadoop分布式集群部署(单namenode节点)详解大数据
- Hadoop知识点总结详解大数据
- hbase错误:Org.apache.hadoop.ipc.RemoteException:User:client is not allowed to impersonate root详解大数据
- Hadoop hdfs完全分布式搭建教程详解大数据
- Hadoop 3.x 新特性剖析系列1详解大数据
- 高可用Hadoop平台-HBase集群搭建详解大数据
- Oracle数据导入Hadoop HDFS(oracle导入hdfs)
- Hadoop——Yarn的HA模式搭建以及WordCount实验详解编程语言
- Linux下快速启动Hadoop集群(linux启动hadoop)
- Debian下安装配置 Hadoop 3.1.3 集群
- Redis与Hadoop:让数据变得更加简单(redis与hadoop)
- adoop数据处理利器:SQL Server与Hadoop(sqlserver.h)