zl程序教程

您现在的位置是:首页 >  工具

当前栏目

Retrying task after failure: Waiting for lock. org.apache.iceberg.hive.HiveTableOperations$WaitingForLockException: Waiting for lock.

Apache for hive org lock Task After failure
2023-09-14 08:57:20 时间

执行spark任务居然碰到了hive中表的被锁了,导致写数据时失败,这个还真是头一次碰到过这回事。

错误信息如下:

22/09/04 21:02:28 WARN Tasks: Retrying task after failure: Waiting for lock.
org.apache.iceberg.hive.HiveTableOperations$WaitingForLockException: Waiting for lock.
        at org.apache.iceberg.hive.HiveTableOperations.lambda$acquireLock$9(HiveTableOperations.java:460)
        at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404)
        at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214)
        at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198)
        at org.apache.iceberg.hive.HiveTableOperations.acquireLock(HiveTableOperations.java:454)
        at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:226)
        at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:127)
        at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:304)
        at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404)
        at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214)
        at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198)
        at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190)
        at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:282)
        at org.apache.iceberg.BaseReplacePartitions.commit(BaseReplacePartitions.java:26)
        at org.apache.iceberg.spark.source.Writer.commitOperation(Writer.java:144)
        at org.apache.iceberg.spark.source.Writer.replacePartitions(Writer.java:181)
        at org.apache.iceberg.spark.source.Writer.commit(Writer.java:119)
        at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec.doExecute(WriteToDataSourceV2Exec.scala:76)
 

跑到hive的元数据库中查询时,在表HIVE_LOCKS中找到了一些信息。
image

根据出错信息,根据hl_table找到相关的表,查看最近的心跳时间字段,hl_last_heartbeat。
删除对应的记录即可。

delete  from HIVE_LOCKS where hl_table = '出错的表名';