记录 hbase:meta,,1.1588230740 is NOT online； state={1588230740 state=OPEN, ts=162
HBase错误解决
在IDEA中远程连接时提示错误
2019-10-08 15:17:28 DEBUG org.apache.zookeeper.ClientCnxn Reading reply sessionid:0x10000192d2c000b, packet:: clientPath:/hbase/meta-region-server serverPath:/hbase/meta-region-server finished:false header:: 11,4 replyHeader:: 11,94489280688,0 request:: '/hbase/meta-region-server,F response:: #ffffffff000146d61737465723a3136303030ffffff9b44fffffff0ffffff8b5f48ffffffbbffffffbc50425546a12a6736c6176653210ffffff947d18ffffffffffffffbfffffff8cffffffb0ffffffcf2d100183,s{34359738425,60129596943,1561451869773,1569062411965,17,0,0,0,53,0,34359738425}
2019-10-08 15:17:28 DEBUG org.apache.zookeeper.ClientCnxn Got ping response for sessionid: 0x10000192d2c000b after 2ms
2019-10-08 15:17:28 DEBUG org.apache.hadoop.hbase.client.RpcRetryingCallerImpl Call exception, tries=9, retries=16, started=29249 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,1 is not online on slave2,16020,1570518833642
…
, details=row ‘user_table’ on table ‘hbase:meta’ at region=hbase:meta,1.1588230740, hostname=slave2,16020,1567495495679, seqNum=-1, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,1 is not online on slave2,16020,1570518833642
…
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.NotServingRegionException): org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,1 is not online on slave2,16020,1570518833642
…
接着在hbase shell中尝试创建表失败,错误提示:
ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2946)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1942)
at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:603)
at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService
2.
c
a
l
l
B
l
o
c
k
i
n
g
M
e
t
h
o
d
(
M
a
s
t
e
r
P
r
o
t
o
s
.
j
a
v
a
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
b
a
s
e
.
i
p
c
.
R
p
c
S
e
r
v
e
r
.
c
a
l
l
(
R
p
c
S
e
r
v
e
r
.
j
a
v
a
:
413
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
b
a
s
e
.
i
p
c
.
C
a
l
l
R
u
n
n
e
r
.
r
u
n
(
C
a
l
l
R
u
n
n
e
r
.
j
a
v
a
:
130
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
b
a
s
e
.
i
p
c
.
R
p
c
E
x
e
c
u
t
o
r
2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor
2.callBlockingMethod(MasterProtos.java)atorg.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)atorg.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)atorg.apache.hadoop.hbase.ipc.RpcExecutorHandler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
查看hbase log文件,错误提示:
2019-10-08 15:41:19,966 WARN [Thread-14] master.HMaster: hbase:meta,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1570520219373, server=slave2,16020,1567495495679}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined; operator intervention required. Schedule an assign.
2019-10-08 15:45:35,967 WARN [Thread-14] master.HMaster: hbase:meta,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1570520219373, server=slave2,16020,1567495495679}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined; operator intervention required. Schedule an assign.
解决办法
进入zookeeper目录
/home/hadoop/zookeeper/apache-zookeeper-3.5.5/bin
运行zookeeper环境命令
zkCli.sh -server localhost:2181
删除zookeeper下的hbase(会提示可以用deleteall代替,但该命令还是可以用)
rmr /hbase
之后远程IDEA连接以及建表都可以正常使用,原因就是在关闭重启集群前得关闭hadoop\hbase\zookpeer,不然再次启动会有奇奇怪怪的问题.
然后删除zk里面的 /hbase目录