E-MapReduce集群启停HDFS/YARN服务

启停HDFS服务

HDFS服务启停，下面脚本需要在master节点运行，切需要在hdfs账号下。su hdfs

启动HDFS

执行下面脚本

#!/bin/bash

worker_cnt=`cat /etc/hosts | grep emr-worker | grep cluster | wc -l`

master_cnt=1

ha_flag=`grep -r high_availability_enable=true /usr/local/emr/emr-bin/script/`

nn_file=/usr/local/emr/emr-bin/script/hdfs/pre_start.sh

dn_file=/usr/local/emr/emr-bin/script/hdfs/start_up.sh

if [[ ! -z $ha_flag ]];then

 master_cnt=2

 nn_file=/usr/local/emr/emr-bin/script/ha_hdfs/pre_start.sh

 dn_file=/usr/local/emr/emr-bin/script/ha_hdfs/start_up.sh

nn_cmd="export app_yarn_home=/usr/lib/hadoop-current;"\ `cat $nn_file | grep -v echo | grep start namenode | head -n 1 | awk -F " {print $2;}`

dn_cmd="export app_yarn_home=/usr/lib/hadoop-current;"\ `cat $dn_file | grep -v echo | grep start datanode | head -n 1 | awk -F " {print $2;}`

#start namenode

for ((i=1; i =$master_cnt; i++));

 echo master--$i

 echo "$nn_cmd"

 if [ $i -eq 2 ];then

 ssh emr-header-$i "/usr/lib/hadoop-current/bin/hdfs namenode -bootstrapStandby N"

 ssh emr-header-$i "$nn_cmd"

# start datanode

for ((i=1; i =$worker_cnt; i++));

 echo "$dn_cmd"

 ssh emr-worker-$i "$dn_cmd"

done

停止HDFS服务

执行下面脚本

#!/bin/bash

worker_cnt=`cat /etc/hosts | grep emr-worker | grep cluster | wc -l`

master_cnt=1

ha_flag=`grep -r high_availability_enable=true /usr/local/emr/emr-bin/script/`

if [[ ! -z $ha_flag ]];then

 master_cnt=2

nn_cmd=/usr/lib/hadoop-current/sbin/hadoop-daemon.sh stop namenode

dn_cmd=/usr/lib/hadoop-current/sbin/hadoop-daemon.sh stop datanode

#stop namenode

for ((i=1; i =$master_cnt; i++));

 ssh emr-header-$i "$nn_cmd"

# stop datanode

for ((i=1; i =$worker_cnt; i++));

 ssh emr-worker-$i "$dn_cmd"

done

启停YARN服务

启停YARN服务，下面的脚本需要在master节点运行，且需要在hadoop账号下，su hadoop。

启动YARN服务

执行下面脚本

#!/bin/bash

worker_cnt=`cat /etc/hosts | grep emr-worker | grep cluster | wc -l`

master_cnt=1

ha_flag=`grep -r high_availability_enable=true /usr/local/emr/emr-bin/script/`

yarn_file=/usr/local/emr/emr-bin/script/yarn/start_up.sh

if [[ ! -z $ha_flag ]];then

 master_cnt=2

 yarn_file=/usr/local/emr/emr-bin/script/ha_yarn/start_up.sh

rm_cmd="export app_yarn_home=/usr/lib/hadoop-current;"\ `cat $yarn_file | grep -v echo | grep start resourcemanager | head -n 1 | awk -F " {print $2;}`

nm_cmd="export app_yarn_home=/usr/lib/hadoop-current;"\ `cat $yarn_file | grep -v echo | grep start nodemanager | head -n 1 | awk -F " {print $2;}`

#start resourcemanager

for ((i=1; i =$master_cnt; i++));

 ssh emr-header-$i "$rm_cmd"

# start nodemanager

for ((i=1; i =$worker_cnt; i++));

 ssh emr-worker-$i "$nm_cmd"

停止YARN服务

执行下面脚本

#!/bin/bash

worker_cnt=`cat /etc/hosts | grep emr-worker | grep cluster | wc -l`

master_cnt=1

ha_flag=`grep -r high_availability_enable=true /usr/local/emr/emr-bin/script/`

if [[ ! -z $ha_flag ]];then

 master_cnt=2

nn_cmd=/usr/lib/hadoop-current/sbin/hadoop-daemon.sh stop namenode

dn_cmd=/usr/lib/hadoop-current/sbin/hadoop-daemon.sh stop datanode

#stop resourcemanager

for ((i=1; i =$master_cnt; i++));

 ssh emr-header-$i "$nn_cmd"

# stop nodemanager

for ((i=1; i =$worker_cnt; i++));

 ssh emr-worker-$i "$dn_cmd"

done

大数据知识面试题-MapReduce和YARN Application Submission Context发出响应，其中包含有：ApplicationID，用户名，队列以及其他启动ApplicationMaster的信息，Container Launch Context（CLC）也会发给ResourceManager，CLC提供了资源的需求，作业文件，安全令牌以及在节点启动ApplicationMaster所需要的其他信息。
一幅长文细学华为MRS大数据开发（五）—— MapReduce和Yarn 本文中主要讲述大数据领域中最著名的批处理和离线处理计算框架——MapReduce，包括MapReduce的原理、流程、使用场景，以及Hadoop集群中负责统一的资源管理和调度的组件——Yarn。
【Hadoop】YARN伪分布式部署和MapReduce案例 前几篇文章 我们介绍了HDFS组件的配置及启动，Yarn是Hadoop集群的资源与作业调度平台，下面介绍下Yarn的伪分布部署及MapReduce简单使用。
nameNode是主节点，datanodes是子节点。子节点之间双重备份。例如小黄点。一共有七个。看上图右下角rep。 HDFS存储方式 是以块的形式存储的，128M最小单元。
阿里云EMR是云原生开源大数据平台，为客户提供简单易集成的Hadoop、Hive、Spark、Flink、Presto、ClickHouse、StarRocks、Delta、Hudi等开源大数据计算和存储引擎，计算资源可以根据业务的需要调整。EMR可以部署在阿里云公有云的ECS和ACK平台。