It is important to note that high I/O wait percentages are not necessarily a problem, especially on machines running I/O bound applications. Large I/O wait percentages can, however, be an indicator of poor I/O performance, or of a hardware or kernel problem.

The only real way to reduce I/O wait percentages is to find and eliminate the bottlenecks that lead to processes spending time in D state. Performance tuning is a broad subject, however, and is outside the scope of this article.

In rare cases, processes can get “stuck” in D state. This can occur due to hardware problems (the kernel is waiting for something from a device that never comes) or from kernel related issues (driver bugs that cause a system call to never return).

Additionally, processes accessing hard NFS mounts will also be put into D state while they wait for the I/O to return - thus a hard NFS mount from a machine that has dropped off of the network can cause processes to go into D state.

From: Red Hat Knowledgebase: What can be done about high I/O wait percentages?