测试Hive集成Sentry

本文在安装和配置Sentry基础之上测试Hive集成Sentry。注意：这里Hive中并没有配置Kerberos认证。

关于配置了Kerberos的Hive集群如何集成Sentry，请参考配置安全的Hive集群集成Sentry。

1. 配置Sentry

见安装和配置Sentry。

2. 配置Hive Hive Metastore集成Sentry

需要在 /etc/hive/conf/hive-site.xml中添加：

 property 

 name hive.metastore.pre.event.listeners /name 

 value org.apache.sentry.binding.metastore.MetastoreAuthzBinding /value 

 /property 

 property 

 name hive.metastore.event.listeners /name 

 value org.apache.sentry.binding.metastore.SentryMetastorePostEventListener /value 

 /property 

Hive-server2集成Sentry

修改 /etc/hive/conf/hive-site.xml，添加以下内容：

 property 

 name hive.server2.enable.impersonation /name 

 value true /value 

 /property 

 property 

 name hive.security.authorization.task.factory /name 

 value org.apache.sentry.binding.hive.SentryHiveAuthorizationTaskFactoryImpl /value 

 /property 

 property 

 name hive.server2.session.hook /name 

 value org.apache.sentry.binding.hive.HiveAuthzBindingSessionHook /value 

 /property 

 property 

 name hive.sentry.conf.url /name 

 value file:///etc/hive/conf/sentry-site.xml /value 

 /property 

参考模板sentry-site.xml.hive-client.template在 /etc/hive/conf/ 目录创建 sentry-site.xml：

 ?xml version="1.0" encoding="UTF-8"? 

 configuration 

 property 

 name sentry.service.client.server.rpc-port /name 

 value 8038 /value 

 /property 

 property 

 name sentry.service.client.server.rpc-address /name 

 value cdh1 /value 

 /property 

 property 

 name sentry.service.client.server.rpc-connection-timeout /name 

 value 200000 /value 

 /property 

 !--以下是客户端配置-- 

 property 

 name sentry.provider /name 

 value org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider /value 

 /property 

 property 

 name sentry.hive.provider.backend /name 

 value org.apache.sentry.provider.db.SimpleDBProviderBackend /value 

 /property 

 property 

 name sentry.metastore.service.users /name 

 value hive /value !--queries made by hive user (beeline) skip meta store check-- 

 /property 

 property 

 name sentry.hive.server /name 

 value server1 /value 

 /property 

 property 

 name sentry.hive.testing.mode /name 

 value true /value 

 /property 

 /configuration 

3. 重启Hive

在cdh1上启动或重启hiveserver2：

$ /etc/init.d/hive-server2 restart

4. 准备测试数据

参考 Securing Impala for analysts，准备测试数据：

$ cat /tmp/events.csv

10.1.2.3,US,android,createNote

10.200.88.99,FR,windows,updateNote

10.1.2.3,US,android,updateNote

10.200.88.77,FR,ios,createNote

10.1.4.5,US,windows,updateTag

然后，在hive中运行下面 sql 语句：

create database sensitive;

create table sensitive.events (

 ip STRING, country STRING, client STRING, action STRING

 ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ,;

load data local inpath /tmp/events.csv overwrite into table sensitive.events;

create database filtered;

create view filtered.events as select country, client, action from sensitive.events;

create view filtered.events_usonly as select * from filtered.events where country = US;

在 cdh1上通过 beeline 连接 hiveserver2，运行下面命令创建角色和组：

# 注意：我设置了hiveserver2的jdbc端口为10001

$ beeline -u "jdbc:hive2://cdh1:10001/" -n hive -p hive -d org.apache.hive.jdbc.HiveDriver

执行下面的 sql 语句创建 role、group等：

create role admin_role;

GRANT ALL ON SERVER server1 TO ROLE admin_role;

GRANT ROLE admin_role TO GROUP admin;

GRANT ROLE admin_role TO GROUP hive;

create role test_role;

GRANT ALL ON DATABASE filtered TO ROLE test_role;

GRANT ROLE test_role TO GROUP test;

上面创建了两个角色：

admin_role，具有管理员权限，可以读写所有数据库，并授权给 admin 和 hive 组（对应操作系统上的组） test_role，只能读写 filtered 数据库，并授权给 test 组。

因为系统上没有test用户和组，所以需要手动创建：

$ useradd test

5. 测试 测试admin_role角色

使用hive用户访问beeline：

$ beeline -u "jdbc:hive2://cdh1:10000/" -n hive -p hive -d org.apache.hive.jdbc.HiveDriver

查看当前系统用户是谁：

0: jdbc:hive2://cdh1:10000/ set system:user.name;

+------------------------+--+

| set |

+------------------------+--+

| system:user.name=hive |

+------------------------+--+

1 row selected (0.188 seconds)

hive属于admin_role组，具有管理员权限，可以查看所有角色：

0: jdbc:hive2://cdh1:10000/ show roles;

+-------------+--+

| role |

+-------------+--+

| test_role |

| admin_role |

+-------------+--+

2 rows selected (0.199 seconds)

查看所有权限：

0: jdbc:hive2://cdh1:10000/ SHOW GRANT ROLE test_role;

+-----------+--------+------------+---------+-----------------+-----------------+------------+---------------+---------------+

| database | table | partition | column | principal_name | principal_type | privilege | grant_option | grant_time | 

+-----------+--------+------------+---------+-----------------+-----------------+------------+---------------+---------------+

| filtered | | | | test_role | ROLE | * | false | 1430293474047000 | 

+-----------+--------+------------+---------+-----------------+-----------------+------------+---------------+---------------+

0: jdbc:hive2://cdh1:10000/ SHOW GRANT ROLE admin_role;

+-----------+--------+------------+---------+-----------------+-----------------+------------+---------------+---------------+

| database | table | partition | column | principal_name | principal_type | privilege | grant_option | grant_time | 

+-----------+--------+------------+---------+-----------------+-----------------+------------+---------------+---------------+

| * | | | | admin_role | ROLE | * | false | 1430293473308000 

+-----------+--------+------------+---------+-----------------+-----------------+------------+---------------+---------------+

1 row selected (0.16 seconds)

hive用户可以查看所有数据库、访问所有表：

0: jdbc:hive2://cdh1:10000/ show databases;

+----------------+--+

| database_name |

+----------------+--+

| default |

| filtered |

| sensitive |

+----------------+--+

3 rows selected (0.391 seconds)

0: jdbc:hive2://cdh1:10000/ use filtered;

No rows affected (0.101 seconds)

0: jdbc:hive2://cdh1:10000/ select * from filtered.events;

+-----------------+----------------+----------------+--+

| events.country | events.client | events.action |

+-----------------+----------------+----------------+--+

| US | android | createNote |

| FR | windows | updateNote |

| US | android | updateNote |

| FR | ios | createNote |

| US | windows | updateTag |

+-----------------+----------------+----------------+--+

5 rows selected (0.431 seconds)

0: jdbc:hive2://cdh1:10000/ select * from sensitive.events;

+---------------+-----------------+----------------+----------------+--+

| events.ip | events.country | events.client | events.action |

+---------------+-----------------+----------------+----------------+--+

| 10.1.2.3 | US | android | createNote |

| 10.200.88.99 | FR | windows | updateNote |

| 10.1.2.3 | US | android | updateNote |

| 10.200.88.77 | FR | ios | createNote |

| 10.1.4.5 | US | windows | updateTag |

+---------------+-----------------+----------------+----------------+--+

5 rows selected (0.247 seconds)

测试test_role角色

使用test用户访问beeline：

$ beeline -u "jdbc:hive2://cdh1:10000/" -n test -p test -d org.apache.hive.jdbc.HiveDriver

查看当前系统用户是谁：

0: jdbc:hive2://cdh1:10000/ set system:user.name;

+------------------------+--+

| set |

+------------------------+--+

| system:user.name=hive |

+------------------------+--+

1 row selected (0.188 seconds)

test用户不是管理员，是不能查看所有角色的：

0: jdbc:hive2://cdh1:10000/ show roles;

ERROR : Error processing Sentry command: Access denied to test. Server Stacktrace: org.apache.sentry.provider.db.SentryAccessDeniedException: Access denied to test

test用户可以列出所有数据库：

0: jdbc:hive2://cdh1:10000/ show databases;

+----------------+--+

| database_name |

+----------------+--+

| default |

| filtered |

| sensitive |

+----------------+--+

3 rows selected (0.079 seconds)

test用户可以filtered库：

0: jdbc:hive2://cdh1:10000/ use filtered;

No rows affected (0.206 seconds)

0: jdbc:hive2://cdh1:10000/ select * from events;

+-----------------+----------------+----------------+--+

| events.country | events.client | events.action |

+-----------------+----------------+----------------+--+

| US | android | createNote |

| FR | windows | updateNote |

| US | android | updateNote |

| FR | ios | createNote |

| US | windows | updateTag |

+-----------------+----------------+----------------+--+

5 rows selected (0.361 seconds)

但是，test用户没有权限访问sensitive库：

0: jdbc:hive2://cdh1:10000/ use sensitive;

Error: Error while compiling statement: FAILED: SemanticException No valid privileges

 Required privileges for this query: Server=server1- Db=sensitive- Table=*- action=insert;Server=server1- Db=sensitive- Table=*- action=select; (state=42000,code=40000)

6. 排错

在CDH5的高版本中，hive cli 不建议使用，在hive集成sentry之后，再运行hive cli 会提示找不到sentry的类的遗产，解决办法是，将sentry相关的jar包链接到hive的home目录下的lib目录下：

ln -s /usr/lib/sentry/lib/sentry-binding-hive.jar /usr/lib/hive/lib/

ln -s /usr/lib/sentry/lib/sentry-core-common.jar /usr/lib/hive/lib

ln -s /usr/lib/sentry/lib/sentry-core-common-db.jar /usr/lib/hive/lib

ln -s /usr/lib/sentry/lib/sentry-policy-common.jar /usr/lib/hive/lib

ln -s /usr/lib/sentry/lib/sentry-policy-db.jar /usr/lib/hive/lib

ln -s /usr/lib/sentry/lib/sentry-policy-cache.jar /usr/lib/hive/lib

ln -s /usr/lib/sentry/lib/sentry-provider-common.jar /usr/lib/hive/lib

ln -s /usr/lib/sentry/lib/sentry-provider-db.jar /usr/lib/hive/lib

ln -s /usr/lib/sentry/lib/sentry-provider-cache.jar /usr/lib/hive/lib

ln -s /usr/lib/sentry/lib/sentry-provider-file.jar /usr/lib/hive/lib

【大数据开发运维解决方案】Sqoop增量同步mysql/oracle数据到hive(merge-key/append）测试文档 上一篇文章介绍了sqoop全量同步数据到hive， 本片文章将通过实验详细介绍如何增量同步数据到hive，以及sqoop job与crontab定时结合无密码登录的增量同步实现方法。
雨客 微博@JavaChen，这里的所有博客文章来自http://blog.javachen.com/。