EMR集群上capacity scheduler的ACL实现

背景

前面一篇介绍了yarn的capacity scheduler原理，实验了在EMR集群上使用capacity scheduler对集群资源的隔离和quota的限制。本文会介绍EMR集群上capacity scheduler的ACL实现。

为什么要做这个？前面给集群分配的资源分配了多个队列，以及每个队列的资源配比和作业调度的优先级。如果多租户里面的每个都按照约定，各自往自己对应的队列里面提交作业，自然没有问题。但是如果用户熟悉capacity scheduler的操作和原理，也是可以占用别组的资源队列。所有有了capacity scheduler的ACL设置。

关键参数

• yarn.scheduler.capacity.queue-mappings

  • 指定用户和queue的映射关系。默认用户上来，不用指定queue参数就能直接到对应的queue。这个比较方便，参数的格式为：[u|g]:[name]:[queue_name][,next mapping]*

• yarn.scheduler.capacity.root.{queue-path}.acl_administer_queue

  • 指定谁能管理这个队列里面的job，英文解释为The ACL of who can administer jobs on the default queue. 星号*表示all，一个空格表示none；

• yarn.scheduler.capacity.root.{queue-path}.acl_submit_applications

  • 指定谁能提交job到这个队列，英文解释是The ACL of who can administer jobs on the queue.星号*表示all，一个空格表示none；

EMR集群上具体操作步骤

• 创建EMR集群

• 修改相关配置来支持queue acl

  • yarn-site: yarn.acl.enable=true
  • mapred-site: mapreduce.cluster.acls.enabled=true
  • hdfs-site: dfs.permissions.enabled=true这个跟capacity scheduler queue的acl没什么关系，是控制hdfs acl的，这里一并设置了
  • hdfs-site: mapreduce.job.acl-view-job=* 如果配置了dfs.permissions.enabled=true，就需要配置一下这个，要不然在hadoop ui上面没发查看job信息

• 重启yarn和hdfs，使配置生效(root账户)

  • su -l hdfs -c '/usr/lib/hadoop-current/sbin/stop-dfs.sh'
  • su -l hadoop -c '/usr/lib/hadoop-current/sbin/stop-yarn.sh'
  • su -l hdfs -c '/usr/lib/hadoop-current/sbin/start-dfs.sh'
  • su -l hadoop -c '/usr/lib/hadoop-current/sbin/start-yarn.sh'
  • su -l hadoop -c '/usr/lib/hadoop-current/sbin/yarn-daemon.sh start proxyserver'
• 修改capacity scheduler配置
  完整配置

<configuration>
  <property>
    <name>yarn.scheduler.capacity.maximum-applications</name>
    <value>10000</value>
    <description>
      Maximum number of applications that can be pending and running.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.25</value>
    <description>
      Maximum percent of resources in the cluster which can be used to run
      application masters i.e. controls number of concurrent running
      applications.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.resource-calculator</name>
    <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>
    <description>
      The ResourceCalculator implementation to be used to compare
      Resources in the scheduler.
      The default i.e. DefaultResourceCalculator only uses Memory while
      DominantResourceCalculator uses dominant-resource to compare
      multi-dimensional resources such as Memory, CPU etc.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>a,b,default</value>
    <description>
      The queues at the this level (root is the root queue).
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.capacity</name>
    <value>20</value>
    <description>Default queue target capacity.</description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.a.capacity</name>
    <value>30</value>
    <description>Default queue target capacity.</description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.b.capacity</name>
    <value>50</value>
    <description>Default queue target capacity.</description>
  </property>


  <property>
    <name>yarn.scheduler.capacity.root.default.user-limit-factor</name>
    <value>1</value>
    <description>
      Default queue user limit a percentage from 0.0 to 1.0.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
    <value>100</value>
    <description>
      The maximum capacity of the default queue.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.state</name>
    <value>RUNNING</value>
    <description>
      The state of the default queue. State can be one of RUNNING or STOPPED.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.a.state</name>
    <value>RUNNING</value>
    <description>
      The state of the default queue. State can be one of RUNNING or STOPPED.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.b.state</name>
    <value>RUNNING</value>
    <description>
      The state of the default queue. State can be one of RUNNING or STOPPED.
    </description>
  </property>


  <property>
    <name>yarn.scheduler.capacity.root.acl_submit_applications</name>
     <value> </value>
     <description>
       The ACL of who can submit jobs to the root queue.
     </description>
   </property>

  <property>
    <name>yarn.scheduler.capacity.root.a.acl_submit_applications</name>
    <value>root</value>
    <description>
      The ACL of who can submit jobs to the default queue.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.b.acl_submit_applications</name>
    <value>hadoop</value>
    <description>
      The ACL of who can submit jobs to the default queue.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
    <value>root</value>
    <description>
      The ACL of who can submit jobs to the default queue.
    </description>
  </property>

<property>
    <name>yarn.scheduler.capacity.root.acl_administer_queue</name>
    <value> </value>
    <description>
      The ACL of who can administer jobs on the default queue.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
    <value>root</value>
    <description>
      The ACL of who can administer jobs on the default queue.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.a.acl_administer_queue</name>
    <value>root</value>
    <description>
      The ACL of who can administer jobs on the default queue.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.b.acl_administer_queue</name>
    <value>root</value>
    <description>
      The ACL of who can administer jobs on the default queue.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.node-locality-delay</name>
    <value>40</value>
    <description>
      Number of missed scheduling opportunities after which the CapacityScheduler
      attempts to schedule rack-local containers.
      Typically this should be set to number of nodes in the cluster, By default is setting
      approximately number of nodes in one rack which is 40.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.queue-mappings</name>
    <value>u:hadoop:b,u:root:a</value>
  </property>

  <property>
    <name>yarn.scheduler.capacity.queue-mappings-override.enable</name>
    <value>false</value>
    <description>
      If a queue mapping is present, will it override the value specified
      by the user? This can be used by administrators to place jobs in queues
      that are different than the one specified by the user.
      The default is false.
    </description>
  </property>

</configuration>

上面的配置，分配了三个队列和对应的资源配比，设置用户hadoop默认（不指定队列的时候）往b队列提，root默认往a队列提。同时hadoop只能往b队列提交作业，root可以往所有队列提交作业。其它用户没有权限提交作业。

踩过的坑

• acl_administer_queue的配置

  • 配置中支持两种操作的acl权限配置acl_administer_queue和acl_submit_applications。按照语意，如果要控制是否能提交作业，只要配置队列的acl_submit_applications属性即可，按照文档，也就是这个意思。但是其实不是的，只要有administer权限的，就能提交作业。这个问题查了好久，找源码才找到。

  @Override
  public void submitApplication(ApplicationId applicationId, String userName,
      String queue)  throws AccessControlException {
    // Careful! Locking order is important!

    // Check queue ACLs
    UserGroupInformation userUgi = UserGroupInformation.createRemoteUser(userName);
    if (!hasAccess(QueueACL.SUBMIT_APPLICATIONS, userUgi)
        && !hasAccess(QueueACL.ADMINISTER_QUEUE, userUgi)) {
      throw new AccessControlException("User " + userName + " cannot submit" +
          " applications to queue " + getQueuePath());
    }

• root queue的配置

  • 如果要限制用户对queue的权限root queue一定要设置，不能只设置leaf queue。因为权限是根权限具有更高的优先级，看代码注释说:// recursively look up the queue to see if parent queue has the permission。这个跟常人理解也不一样。所以需要先把把的权限限制住，要不然配置的各种自队列的权限根本没有用。

<property>
    <name>yarn.scheduler.capacity.root.acl_submit_applications</name>
     <value> </value>
     <description>
       The ACL of who can submit jobs to the root queue.
     </description>
   </property>