POSIX AIO, Libio & DIRECT-IO

POSIX AIO vs libaio

  1. The POSIX AIO is a user-level implementation that performs normal blocking I/O in multiple threads. (-lrt)
  2. The kernel AIO (i.e. io_submit() et.al.) is kernel support for asynchronous I/O operations, where the io requests are actually queued up in the kernel. (-laio)

参考: Difference between POSIX AIO and libaio on Linux?

The current Linux POSIX AIO implementation is provided in user space by glibc.
This has a number of limitations, most notably that maintaining multiple threads to perform I/O operations is expensive and scales poorly. Work has been in progress for some time on a kernel state-machine-based implementation of asynchronous I/O.

libaio 分析

libaio 是 io_submit() 等系统调用的封装. 一般只用在存储设备的读写操作上,如磁盘,闪存盘等。经常与 O_DIRECT 配合使用。

Currently, the AIO interface is best for O_DIRECT access to a raw block device like a disk, flash drive or storage array.

What Works?

  • AIO read and write on raw (and O_DIRECT on blockdev).
  • AIO read and write on files opened with O_DIRECT on ext2, ext3, jfs, xfs.

What Does Not Work?

  • AIO read and write on files opened without O_DIRECT (i.e. normal buffered filesystem AIO). On ext2, ext3, jfs, xfs and nfs, these do not return an explicit error, but quietly default to synchronous or rather non-AIO behaviour (i.e io_submit waits for I/O to complete in these cases). For most other filesystems, -EINVAL is reported.
  • AIO fsync (not supported for any filesystem).
  • AIO read and write on sockets (doesn’t return an explicit error, but quietly defaults to synchronous or rather non-AIO behavior).
  • AIO read and write on pipes (reports -EINVAL).
  • Not all devices (including TTY) support AIO (typically return -EINVAL).

参考:

  1. Kernel Asynchronous I/O (AIO) Support for Linux
  2. http://www.linuxsymposium.org/archives/OLS/Reprints-2002/lahaise-reprint.pdf
  3. Linux Asynchronous I/O Explained

libaio DEMO

  1. AIOUserGuide
  2. Linux下libaio的一个简单例子

Direct I/O

Direct I/O is a feature of the file system whereby file reads and writes go directly from the applications to the storage device, bypassing the operating system read and write caches. Direct I/O is used only by applications (such as databases) that manage their own caches.

An application invokes direct I/O by opening a file with the O_DIRECT flag. Alternatively, GFS can attach a direct I/O attribute to a file, in which case direct I/O is used regardless of how the file is opened.

When a file is opened with O_DIRECT, or when a GFS direct I/O attribute is attached to a file, all I/O operations must be done in block-size multiples of 512 bytes. The memory being read from or written to must also be 512-byte aligned.

To cause the O_DIRECT flag to be defined with recent glibc libraries, define _GNU_SOURCE at the beginning of a source file before any includes, or define it on the cc line when compiling.

参考:

  1. What is Direct I/O
  2. 3.9. Direct I/O
  3. Direct I/O
  4. direct-io

Comments