Data Sharing Made Easier through Programmable Metadata Zhe Zhang IBM Research

!

Remzi Arpaci-Dusseau University of Wisconsin-Madison

How do applications share data today? –Syncing data between storage systems: • Commonly used big data workflow • Slow, stale and strenuous

Cloud analytics cluster

Primary Data: transactions, emails, logs, etc. In-house analytics cluster

!2

How do applications share data today? –Syncing data between storage systems: • Commonly used big data workflow • Slow, stale and strenuous

Cloud analytics cluster

Primary Data: transactions, emails, logs, etc. In-house analytics cluster

!2

– Mounting and using shared storage systems: • Difficult to serve heterogenous workloads • Heavy workload on centralized name nodes

How do applications share data today? –Syncing data between storage systems: • Commonly used big data workflow • Slow, stale and strenuous

– Mounting and using shared storage systems: • Difficult to serve heterogenous workloads • Heavy workload on centralized name nodes

Cloud analytics cluster

Primary Data: transactions, emails, logs, etc. In-house analytics cluster

Observations –Data always written and read through the same storage system (filesystem, DB, etc.) • Metadata updated with writes • Metadata used in reads –Data produced in form A and consumed in form B? • View DB records as a file? • Analyze thousands of local log files as a single text file?

!2

How do applications share data today? –Syncing data between storage systems: • Commonly used big data workflow • Slow, stale and strenuous

– Mounting and using shared storage systems: • Difficult to serve heterogenous workloads • Heavy workload on centralized name nodes

Cloud analytics cluster

Primary Data: transactions, emails, logs, etc. In-house analytics cluster

Observations –Data always written and read through the same storage system (filesystem, DB, etc.) • Metadata updated with writes • Metadata used in reads –Data produced in form A and consumed in form B? • View DB records as a file? • Analyze thousands of local log files as a single text file?

!2

Programming the Metadata segment 1

Source file 1

Logical definition Under the hood

!3

segment 2

segment 3

Source file 2 Source DB table

Programming the Metadata segment 1

Source file 1

segment 2

segment 3

Source file 2

Logical definition Under the hood

Source DB table

Source file 1

length atime mtime … block

length atime mtime … block Source file 2

block …

!3

blk blk

block blk blk

Programming the Metadata segment 1

Source file 1

segment 2

segment 3

Source file 2

Logical definition Under the hood

Source DB table

Source file 1

length

length

atime

atime

mtime … block length atime mtime … block Source file 2

block …

!3

mtime blk blk

… block

block

block blk blk

block block

Programming the Metadata segment 1

Source file 1

segment 2

segment 3

Source file 2

Logical definition Under the hood

Source DB table

Source file 1

length

length

atime

atime

mtime … block length

block

atime

block

mtime … block Source file 2

block …

!3

mtime blk blk blk

… block block

blk blk

block block block

Challenges § API challenge: identification / namespace of source data – How to define a file in VM1 to include a source file in VM2? – Granularity-based source file selection: 1 out of 10 lines of text? – Content-based source file selection: all lines containing certain keyword? – Arbitrary “SELECT * FROM * WHERE *” in source DB tables?

§ Performance challenge: frequent metadata updates

!4

Layers

Example of Liseners

Applications

• Map to destination file if keyword matches • Map every 1 line out of 10 lines of text to destination file

VFS

• Map entire file to destination file • Map every 1MB out of 10MB to destination file

Block storage

• All VFS listeners can be implemented on block layer with a reverse pointer from block to inode

Use Case: Distributed Live Analytics § hadoop dfs -composeFromLocal § Configuration file slave1:/opt/IBM/*/*.log slave2:/var/*.log …

§ Challenges – Informing NameNode of local file size changes – Balancing workload Email Server VM

Email Server VM

New York

Wed Server

San Jose

Wed Server VM

VM

Raleigh Dallas

EmailServerLogs !5

EasternCoastLogs

MapReduce, Stream, etc.