Async buffered disk IO for userspace apps

Milosz Tanski CTO @ Adfin

Use cases / Background ●

Network applications that cannot use sendfile()

●

Async disk IO is O_DIRECT only

●

Not for a lack of trying... 10 years of attempts

●

Userspace solution = IO threadpools

IO thread pool pattern

IO thread pool pattern ●

IO threadpools increase request latency –

Synchronization

–

Migrating threads / cores

–

Fast (cached data) stuck behind slow requests

preadv2 syscall ssize_t preadv2(int fd, const struct iovec *iov, int iovcnt, off_t offset, int flags); flags: RWF_NONBLOCK ●

Read data only if in pagecache or EAGAIN

●

Skip queue for “fast reads“ in network loop

●

Minimal change to userspace app

Tests / numbers ●

Developed a Samba FIO engine MB/s

c. iops

mean latency (us)

SYNC

151.47

38,775

24.45

TP

112.68

28,845

33.25

RWF_NONBLOCK

140.04

35,850

26.54

4K rand read / all cached

64kb seq read / 16gig file / 2gb ram / 5400k rpm SYNC

517.64

8,282

119.87

TP

412.42

6,598

150.72

RWF_NONBLOCK

483.94

7,742

128.37

2mb seq read / 16gig file / 2gb ram / 5400k rpm SYNC

508.42

254

3931.82

TP

390.22

195

5123.45

RWF_NONBLOCK

473.41

236

4223.19

Summary ●

Our app (column OLAP) –

Magnitude large data then memory

–

Hierarchical storage: ●

–

Pagecache → FSCache (ssd) → cephfs (hdd / tiered)

23% lower response times

●

Works well with sequential / zipf workflows

●

Threadpools are expensive

Discussion and feedback ●

LWN article: –

●

Latest patchset (v6): –

●

https://lwn.net/Articles/612483/ https://lkml.org/lkml/2014/11/10/466

Man pages patches (v2): –

https://lkml.org/lkml/2014/11/6/447