How to kill an `uninterruptible sleep` process
很早之前就遇到一次这个故障,当时是一台主机故障,这次是上百台主机故障。当时是使用mysqldump向NFS备份时,写数据时大概是NFS出现故障,使得mysqldump进程进入uninterruptible sleep(man ps)状态:
$ps axu|grep mysqldumpmysql 2718 0.0 0.0 51088 672 pts/0 S+ 13:30 0:00 grep mysqldump
mysql 14916 1.4 0.0 0 0 ? D 02:03 10:03 [mysqldump]
进入该状态的进程,会一直等待NFS,不接受任何信号,当然也就无法被杀死(kill/fuser -k)。因为进程一直在运行队列(running queue)中,所以还会导致主机的Load上升(虽然主机并不繁忙)。如果由于这个原因被卡住的进程很多的话,主机的Load可能会看起来非常高。
上次出现这个问题时只有一台主机出了这个症状,当时Google后也注意到很多人有遇到了类似的问题,并且都束手无策(There are some things even root cant do)。Google后,再尝试找了一些Linux原理方面的介绍,依然无解。
解决的终极办法是重启主机(reboot),所以决定等待一段时间(之前遇到过一些僵死进程Z,等待一段时间后也消失了),实在不行就择机重启主机。不过奇怪的是,等了几天过后,这个进程确实消失了(正常结束了?应该是)。
1. Why is Linux so "stupid"?
有人就问,Linux设计是不是有问题,为什么会有一些进程root也无法杀死呢?
jra在How to kill a process in uninterruptible sleep state中给了一个解释:
Theres fairly extensive discussion of this in a couple of the kernel design books, and I think in Nemeth, Snyder and Seebass: the problem stems from the fact that there are two types of device drivers -- those for "fast" devices and those for "slow" devices: Slow-device drivers -- for things like terminals, and such -- are usually split in two pieces, and can therefore be interrupted while theyre in the middle of something. Fast-device drivers -- which service things like hard drives and (I think) ethernet cards -- are designed to expect that when they call out to hardware, it will respond instantly (in human terms), and that they wont have to wait on anything. Such drivers have, as a rule, proven extremely intolerant of hardware trouble -- if your hard drive start having to do hardware retries to read a sector, your system perfromance is going int he toilet, even if you have more than one drive...
大概意思是说,Linux中设备驱动程序可以分为“slow device”和“fast device”两类。磁盘属于“fast device”(如果当作“slow device”处理效率会很低),在这类设备上操作时会以uninterruptible的方式进行。
2. 真的没办法吗?
理论上,除了重启主机,貌似没什么办法了。不过,根据实际经验和运气,也还是有一些办法可以尝试的。
办法1:umount -f
如果是由于NFS故障导致的,可以尝试使用先umount/mount重新挂载NFS。如果NFS无法卸载(如果遇到上述情况,八成是这样),可以尝试使用 -f 参数卸载。最近一次遇到这个故障,有接近上百台主机出现这个故障,就是使用umount -f成功卸载NFS后,相关uninterruptible的进程也都随之正常结束。当时发现,连续重试多次 umount -f 才行:
root #umount -f /nfsdirumount2: Device or resource busy
umount: /nfsdir: device is busy
root #umount -f /nfsdir
umount2: Device or resource busy
umount: /nfsdir: device is busy
办法2:等
最早mysqldump遇到这个问题时,束手无策后,就等了几天,发现进程确实结束了。所以如果情况不是很紧急,“等”也是一个办法。
遇到的僵死进程这个办法也可能有效。
办法3:killall -KILL rpciod
在一篇文章中,还提到,进程处于uninterruptible sleep(即ps的D状态),进程处于rpc_execute调用状态,而rpc_execute调用是由rpciod提供,所以可以通过杀死rpciod来解决问题。(rpciod被杀死后会自动重启过来)【没有实验过,慎用,希望有经验者分享一下】
办法4:reboot
实在不行,就只能reboot了。这是最不推荐的做法,毕竟每次遇到都reboot,代价还是比较大,特别是当你在追求HA的时候。
3. 能够避免这种情况吗?
在RTFM之后,发现可以通过挂载NFS时指定一些参数,尽可能的避免这个问题:
soft If an NFS file operation has a major timeout then report an I/O error to the calling program. The default is to continue retrying NFS file operations indefinitely. hard If an NFS file operation has a major timeout then report "server not responding" on the console and continue retrying indefinitely. This is the default. intr If an NFS file operation has a major timeout and it is hard mounted, then allow signals to interupt the file operation and cause it to return EINTR to the calling program. The default is to not allow file operations to be interrupted.
上面3个参数,其中hard是默认的。我们通过断开网络模拟NFS故障,测试了 intr 选项,发现能够一定程度上避免上述的情况。在测试 soft 选项时,并没有上面描述的那么好用,在实验中,使用 soft 挂载是现象和 hard 类似(不知道我是不是忽略哪些细节了)。
如果你被这个问题困扰,可以试试使用intr参数挂载NFS:
mount -o intr 172.23.119.25:/nfs /nfsdirsleep与wait区别 第一个区别是在对系统资源的占用上。 wait是Object类的一个函数(也就意味着所有对象都有这个函数),指线程处于进入等待状态,此时线程不占用任何资源,不增加时间限制。wait可以被notify和notifyAll函数唤醒(当然这两个同时也是Object的函数)。 而sleep则是Thread类的一个函数,指线程被调用时,占着CPU不工作。此时,系统的CPU部分资源被占用,其他线程无法进入,会增加时间限制。
sleep( ) 和 wait( ) 的这 5 个区别,你知道几个? sleep(休眠) 和 wait(等待) 方法是 Java 多线程中常用的两个方法,它们有什么区别及一些该注意的地方有哪些呢?下面给大家一一分解。
关于java线程中stop interrupt daemon wait notify 一。关于终止线程stop与interrupt 一般来说,线程执行结束后就变成消亡状态,乍看之下我们并不需要人为进行干预(人为停止线程),不过凡事都有例外吧,在服务器或者其他应用场景下,线程为了提供服务而一直在不停的运转,因此必要时刻我们还需“人为干涉的”。
相关文章
- Redirecting to /bin/systemctl start mysqld.service Failed to start mysqld.service: Unit not found.
- to program_I Just Want To
- 错误解决:widget.FrameLayout$LayoutParams cannot be cast to android.widget.LinearLayout$LayoutParams
- ORA-24052: cannot propagate object type payloads with LOB attributes to an 8.0 release ORACLE 报错 故障修复 远程处理
- ORA-24322: unable to delete an initialized mutex ORACLE 报错 故障修复 远程处理
- ORA-24901: handles belonging to different environments passed into an OCI call ORACLE 报错 故障修复 远程处理
- ORA-26869: must attach to an XStream inbound server before executing string function ORACLE 报错 故障修复 远程处理
- ORA-26898: Unable to create “string” because there is an Oracle Capture process using the same queue “string”.”string”. ORACLE 报错 故障修复 远程处理
- ORA-28670: mapping table cannot be dropped due to an existing bitmap index ORACLE 报错 故障修复 远程处理
- ORA-29518: name string resolved to an object in schema string that is not a Java class ORACLE 报错 故障修复 远程处理
- ORA-30748: column string already enabled to store objects of type string.string ORACLE 报错 故障修复 远程处理
- ORA-39504: failed to notify CRS of a Startup/Shutdown event [string] (ignored) ORACLE 报错 故障修复 远程处理
- ORA-48116: error enountered when attempting to create a directory [string] ORACLE 报错 故障修复 远程处理
- ORA-56926: database must be in UPGRADE mode in order to start an upgrade window ORACLE 报错 故障修复 远程处理
- ORA-01156: recovery or flashback in progress may need access to files ORACLE 报错 故障修复 远程处理
- ORA-13195: failed to generate maximum tile value ORACLE 报错 故障修复 远程处理
- ORA-13203: failed to read USER_SDO_GEOM_METADATA view ORACLE 报错 故障修复 远程处理
- ORA-15188: process terminated due to an ASMLIB error ORACLE 报错 故障修复 远程处理
- ORA-16003: standby database is restricted to read-only access ORACLE 报错 故障修复 远程处理
- ORA-16408: An archive log from an incompatible redo branch has been rejected. ORACLE 报错 故障修复 远程处理
- ORA-16426: Recovery requested an incorrect log from which to apply redo data. ORACLE 报错 故障修复 远程处理
- ORA-16573: attempt to change or access configuration file for an enabled broker configuration ORACLE 报错 故障修复 远程处理
- ORA-16962: SQL statement attempted to begin an autonomous transaction ORACLE 报错 故障修复 远程处理
- ORA-19023: The first argument to UPDATEXML operator has to be an XMLTYPE ORACLE 报错 故障修复 远程处理
- [问题解决]大数据量上载excel文件数据到SAP系统[ALSM_EXCEL_TO_INTERNAL_TABLE]详解编程语言
- Powering Your System: An Introduction to the Linux Ash Shell(linuxash)
- Unlocking the Power of Oracle 11: Your Ultimate Guide to a Successful Database Management(oracle11个)