EMR集群上capacity scheduler的ACL实现
2023-03-09 22:14:45 时间
背景
前面一篇介绍了yarn的capacity scheduler原理,实验了在EMR集群上使用capacity scheduler对集群资源的隔离和quota的限制。本文会介绍EMR集群上capacity scheduler的ACL实现。
为什么要做这个?前面给集群分配的资源分配了多个队列,以及每个队列的资源配比和作业调度的优先级。如果多租户里面的每个都按照约定,各自往自己对应的队列里面提交作业,自然没有问题。但是如果用户熟悉capacity scheduler的操作和原理,也是可以占用别组的资源队列。所有有了capacity scheduler的ACL设置。
关键参数
-
yarn.scheduler.capacity.queue-mappings
- 指定用户和queue的映射关系。默认用户上来,不用指定queue参数就能直接到对应的queue。这个比较方便,参数的格式为:
[u|g]:[name]:[queue_name][,next mapping]*
- 指定用户和queue的映射关系。默认用户上来,不用指定queue参数就能直接到对应的queue。这个比较方便,参数的格式为:
-
yarn.scheduler.capacity.root.{queue-path}.acl_administer_queue
- 指定谁能管理这个队列里面的job,英文解释为
The ACL of who can administer jobs on the default queue.
星号*
表示all,一个空格表示none;
- 指定谁能管理这个队列里面的job,英文解释为
-
yarn.scheduler.capacity.root.{queue-path}.acl_submit_applications
- 指定谁能提交job到这个队列,英文解释是
The ACL of who can administer jobs on the queue.
星号*
表示all,一个空格表示none;
- 指定谁能提交job到这个队列,英文解释是
EMR集群上具体操作步骤
- 创建EMR集群
-
修改相关配置来支持queue acl
- yarn-site:
yarn.acl.enable=true
- mapred-site:
mapreduce.cluster.acls.enabled=true
- hdfs-site:
dfs.permissions.enabled=true
这个跟capacity scheduler queue的acl没什么关系,是控制hdfs acl的,这里一并设置了 - hdfs-site:
mapreduce.job.acl-view-job=*
如果配置了dfs.permissions.enabled=true
,就需要配置一下这个,要不然在hadoop ui上面没发查看job信息
- yarn-site:
-
重启yarn和hdfs,使配置生效(root账户)
su -l hdfs -c '/usr/lib/hadoop-current/sbin/stop-dfs.sh'
su -l hadoop -c '/usr/lib/hadoop-current/sbin/stop-yarn.sh'
su -l hdfs -c '/usr/lib/hadoop-current/sbin/start-dfs.sh'
su -l hadoop -c '/usr/lib/hadoop-current/sbin/start-yarn.sh'
su -l hadoop -c '/usr/lib/hadoop-current/sbin/yarn-daemon.sh start proxyserver'
- 修改capacity scheduler配置
完整配置
<configuration>
<property>
<name>yarn.scheduler.capacity.maximum-applications</name>
<value>10000</value>
<description>
Maximum number of applications that can be pending and running.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>0.25</value>
<description>
Maximum percent of resources in the cluster which can be used to run
application masters i.e. controls number of concurrent running
applications.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.resource-calculator</name>
<value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>
<description>
The ResourceCalculator implementation to be used to compare
Resources in the scheduler.
The default i.e. DefaultResourceCalculator only uses Memory while
DominantResourceCalculator uses dominant-resource to compare
multi-dimensional resources such as Memory, CPU etc.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.queues</name>
<value>a,b,default</value>
<description>
The queues at the this level (root is the root queue).
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.capacity</name>
<value>20</value>
<description>Default queue target capacity.</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.a.capacity</name>
<value>30</value>
<description>Default queue target capacity.</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.b.capacity</name>
<value>50</value>
<description>Default queue target capacity.</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.user-limit-factor</name>
<value>1</value>
<description>
Default queue user limit a percentage from 0.0 to 1.0.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
<value>100</value>
<description>
The maximum capacity of the default queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.state</name>
<value>RUNNING</value>
<description>
The state of the default queue. State can be one of RUNNING or STOPPED.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.a.state</name>
<value>RUNNING</value>
<description>
The state of the default queue. State can be one of RUNNING or STOPPED.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.b.state</name>
<value>RUNNING</value>
<description>
The state of the default queue. State can be one of RUNNING or STOPPED.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.acl_submit_applications</name>
<value> </value>
<description>
The ACL of who can submit jobs to the root queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.a.acl_submit_applications</name>
<value>root</value>
<description>
The ACL of who can submit jobs to the default queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.b.acl_submit_applications</name>
<value>hadoop</value>
<description>
The ACL of who can submit jobs to the default queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
<value>root</value>
<description>
The ACL of who can submit jobs to the default queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.acl_administer_queue</name>
<value> </value>
<description>
The ACL of who can administer jobs on the default queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
<value>root</value>
<description>
The ACL of who can administer jobs on the default queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.a.acl_administer_queue</name>
<value>root</value>
<description>
The ACL of who can administer jobs on the default queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.b.acl_administer_queue</name>
<value>root</value>
<description>
The ACL of who can administer jobs on the default queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.node-locality-delay</name>
<value>40</value>
<description>
Number of missed scheduling opportunities after which the CapacityScheduler
attempts to schedule rack-local containers.
Typically this should be set to number of nodes in the cluster, By default is setting
approximately number of nodes in one rack which is 40.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.queue-mappings</name>
<value>u:hadoop:b,u:root:a</value>
</property>
<property>
<name>yarn.scheduler.capacity.queue-mappings-override.enable</name>
<value>false</value>
<description>
If a queue mapping is present, will it override the value specified
by the user? This can be used by administrators to place jobs in queues
that are different than the one specified by the user.
The default is false.
</description>
</property>
</configuration>
上面的配置,分配了三个队列和对应的资源配比,设置用户hadoop默认(不指定队列的时候)往b队列提,root默认往a队列提。同时hadoop只能往b队列提交作业,root可以往所有队列提交作业。其它用户没有权限提交作业。
踩过的坑
-
acl_administer_queue的配置
- 配置中支持两种操作的acl权限配置
acl_administer_queue
和acl_submit_applications
。按照语意,如果要控制是否能提交作业,只要配置队列的acl_submit_applications
属性即可,按照文档,也就是这个意思。但是其实不是的,只要有administer权限的,就能提交作业。这个问题查了好久,找源码才找到。
- 配置中支持两种操作的acl权限配置
@Override
public void submitApplication(ApplicationId applicationId, String userName,
String queue) throws AccessControlException {
// Careful! Locking order is important!
// Check queue ACLs
UserGroupInformation userUgi = UserGroupInformation.createRemoteUser(userName);
if (!hasAccess(QueueACL.SUBMIT_APPLICATIONS, userUgi)
&& !hasAccess(QueueACL.ADMINISTER_QUEUE, userUgi)) {
throw new AccessControlException("User " + userName + " cannot submit" +
" applications to queue " + getQueuePath());
}
-
root queue的配置
- 如果要限制用户对queue的权限root queue一定要设置,不能只设置leaf queue。因为权限是根权限具有更高的优先级,看代码注释说:
// recursively look up the queue to see if parent queue has the permission
。这个跟常人理解也b不一样。所以需要先把把的权限限制住,要不然配置的各种自队列的权限根本没有用。
- 如果要限制用户对queue的权限root queue一定要设置,不能只设置leaf queue。因为权限是根权限具有更高的优先级,看代码注释说:
<property>
<name>yarn.scheduler.capacity.root.acl_submit_applications</name>
<value> </value>
<description>
The ACL of who can submit jobs to the root queue.
</description>
</property>
相关文章
- 使用全新的 AWS WAF Bot Control 减少您网站上不需要的流量
- 如何开始使用适用于 Amazon VPC 的 Amazon Route 53 Resolver DNS 防火墙
- CloudWatch Metric Streams – 实时将 AWS 指标发送给合作伙伴和您的应用程序
- 排查新版 EC2 串行控制台的启动和联网问题
- PYTHON-FUNCTOOLS模块
- Red Hat OpenShift Service on AWS 现已正式推出
- 数据分析的技术源流
- AWS PrivateLink for Amazon S3 现已全面上线
- 成本分配博文之二:Amazon 生成与用户定义的成本分配标签
- 用 Microsoft ADFS 4.0 AWS 做 AppStream 2.0 单点登录部署
- 成本分配博文系列之一:您需要了解的成本分配基础知识
- 使用Microsoft AppLocker应用程序允许列表管理在Amazon AppStream 2.0上的应用程序
- 如何将开源项目迁移到 SageMaker
- 通过 EventBridge+SNS+Lambda 实时捕获EMR集群的状态变化
- 基于 Comprehend 的玩家评论分析解决方案
- Amazon Elastic File System 新增低成本单区存储类
- 采用 Amazon Transcribe 服务快速为大型线上会议视频添加字幕
- 面向农业科技的 AI:使用 Amazon Rekognition 自定义标签对猕猴桃进行分类
- NoCode 热身系列:实现一个任务跟踪与管理系统(下)
- Python web-Http