On modeling the relative fitness of storage (data appendices) Michael P. Mesnier November 27, 2007
A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy
Thesis committee Prof. Gregory R. Ganger, Chair Prof. Christos Faloutsos Dr. Mic Bowman (Intel Research) Dr. Arif Merchant (Hewlett-Packard Laboratories)
Department of Electrical and Computer Engineering Carnegie Mellon University Pittsburgh, PA 15213
2
Contents A Per-application summary
13
B Mixed-model summary
29
C FitnessDirect model training
45
D FitnessBuffered model training
57
E FitnessFS model training
69
F FitnessCache model training
81
G Postmark model training
93
H Cello model training
105
I
117
TPC-C model training
J WorkloadMix model training
129
K FitnessDirect model testing on FitnessDirect samples
141
L FitnessBuffered model testing on FitnessBuffered samples
151
M FitnessFS model testing on FitnessFS samples
161
N FitnessCache model testing on FitnessCache samples
171
O Postmark model testing on Postmark samples
181
P Cello model testing on Cello samples
191
Q TPC-C model testing on TPC-C samples
201
R WorkloadMix model testing on FitnessDirect samples
211
S WorkloadMix model testing on FitnessBuffered samples
221
T WorkloadMix model testing on FitnessFS samples
231
U WorkloadMix model testing on FitnessCache samples
241
V WorkloadMix model testing on Postmark samples
251
W WorkloadMix model testing on Cello samples
261
3
CONTENTS X WorkloadMix model testing on TPC-C samples
4 271
List of Figures A.1 A.2 A.3 A.4 A.5 A.6 A.7 A.8 A.9
Error vs. training: Per-application summary (Median relative error) Error vs. training: Per-application summary (Mean relative error) . Overall relative error CDFs: Per-application summary . . . . . . . . Bandwidth relative error CDFs: Per-application summary . . . . . . Throughput relative error CDFs: Per-application summary . . . . . Latency relative error CDFs: Per-application summary . . . . . . . . Error reduction PDFs: Per-application summary . . . . . . . . . . . Median relative error: Per-application summary . . . . . . . . . . . . Mean relative error: Per-application summary . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
16 17 18 19 20 21 22 22 26
B.1 B.2 B.3 B.4 B.5 B.6 B.7 B.8 B.9
Error vs. training: Mixed-model summary (Median relative error) Error vs. training: Mixed-model summary (Mean relative error) . Overall relative error CDFs: Mixed-model summary . . . . . . . . Bandwidth relative error CDFs: Mixed-model summary . . . . . . Throughput relative error CDFs: Mixed-model summary . . . . . . Latency relative error CDFs: Mixed-model summary . . . . . . . . Error reduction PDFs: Mixed-model summary . . . . . . . . . . . Median relative error: Mixed-model summary . . . . . . . . . . . . Mean relative error: Mixed-model summary . . . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
32 33 34 35 36 37 38 38 42
C.1 C.2 C.3 C.4 C.5 C.6 C.7 C.8
Performance graphs: FitnessDirect . . . . . . . . . . . . . . . . . . . . Performance PDFs: FitnessDirect . . . . . . . . . . . . . . . . . . . . . . Bandwidth relative fitness PDFs: FitnessDirect . . . . . . . . . . . . . . Throughput relative fitness PDFs: FitnessDirect . . . . . . . . . . . . . Latency relative fitness PDFs: FitnessDirect . . . . . . . . . . . . . . . . Workload characteristic CDFs: FitnessDirect . . . . . . . . . . . . . . . Error vs. training: FitnessDirect model training (Median relative error) Error vs. training: FitnessDirect model training (Mean relative error) .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
47 48 49 50 51 52 55 56
D.1 D.2 D.3 D.4 D.5 D.6 D.7 D.8
Performance graphs: FitnessBuffered . . . . . . . . . . . . . . . . . . . Performance PDFs: FitnessBuffered . . . . . . . . . . . . . . . . . . . . . Bandwidth relative fitness PDFs: FitnessBuffered . . . . . . . . . . . . . . Throughput relative fitness PDFs: FitnessBuffered . . . . . . . . . . . . . Latency relative fitness PDFs: FitnessBuffered . . . . . . . . . . . . . . . Workload characteristic CDFs: FitnessBuffered . . . . . . . . . . . . . . . Error vs. training: FitnessBuffered model training (Median relative error) Error vs. training: FitnessBuffered model training (Mean relative error) .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
59 60 61 62 63 64 67 68
E.1 E.2 E.3 E.4 E.5
Performance graphs: FitnessFS . . . . . . . Performance PDFs: FitnessFS . . . . . . . . Bandwidth relative fitness PDFs: FitnessFS . Throughput relative fitness PDFs: FitnessFS Latency relative fitness PDFs: FitnessFS . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
71 72 73 74 75
5
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
6
LIST OF FIGURES E.6 Workload characteristic CDFs: FitnessFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E.7 Error vs. training: FitnessFS model training (Median relative error) . . . . . . . . . . . . . . E.8 Error vs. training: FitnessFS model training (Mean relative error) . . . . . . . . . . . . . . .
76 79 80
F.1 F.2 F.3 F.4 F.5 F.6 F.7 F.8
Performance graphs: FitnessCache . . . . . . . . . . . . . . . . . . . . Performance PDFs: FitnessCache . . . . . . . . . . . . . . . . . . . . . . Bandwidth relative fitness PDFs: FitnessCache . . . . . . . . . . . . . . Throughput relative fitness PDFs: FitnessCache . . . . . . . . . . . . . Latency relative fitness PDFs: FitnessCache . . . . . . . . . . . . . . . . Workload characteristic CDFs: FitnessCache . . . . . . . . . . . . . . . Error vs. training: FitnessCache model training (Median relative error) Error vs. training: FitnessCache model training (Mean relative error) .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
83 84 85 86 87 88 91 92
G.1 G.2 G.3 G.4 G.5 G.6 G.7 G.8
Performance graphs: Postmark . . . . . . . . . . . . . . . . . . . . . Performance PDFs: Postmark . . . . . . . . . . . . . . . . . . . . . . Bandwidth relative fitness PDFs: Postmark . . . . . . . . . . . . . . Throughput relative fitness PDFs: Postmark . . . . . . . . . . . . . Latency relative fitness PDFs: Postmark . . . . . . . . . . . . . . . . Workload characteristic CDFs: Postmark . . . . . . . . . . . . . . . Error vs. training: Postmark model training (Median relative error) Error vs. training: Postmark model training (Mean relative error) .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. 95 . 96 . 97 . 98 . 99 . 100 . 103 . 104
H.1 H.2 H.3 H.4 H.5 H.6 H.7 H.8
Performance graphs: Cello . . . . . . . . . . . . . . . . . . . . Performance PDFs: Cello . . . . . . . . . . . . . . . . . . . . . Bandwidth relative fitness PDFs: Cello . . . . . . . . . . . . . Throughput relative fitness PDFs: Cello . . . . . . . . . . . . . Latency relative fitness PDFs: Cello . . . . . . . . . . . . . . . Workload characteristic CDFs: Cello . . . . . . . . . . . . . . . Error vs. training: Cello model training (Median relative error) Error vs. training: Cello model training (Mean relative error) .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
107 108 109 110 111 112 115 116
I.1 I.2 I.3 I.4 I.5 I.6 I.7 I.8
Performance graphs: TPC-C . . . . . . . . . . . . . . . . . . . . . . Performance PDFs: TPC-C . . . . . . . . . . . . . . . . . . . . . . Bandwidth relative fitness PDFs: TPC-C . . . . . . . . . . . . . . Throughput relative fitness PDFs: TPC-C . . . . . . . . . . . . . . Latency relative fitness PDFs: TPC-C . . . . . . . . . . . . . . . . Workload characteristic CDFs: TPC-C . . . . . . . . . . . . . . . . Error vs. training: TPC-C model training (Median relative error) . Error vs. training: TPC-C model training (Mean relative error) . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
119 120 121 122 123 124 127 128
J.1 J.2 J.3 J.4 J.5 J.6 J.7 J.8
Performance graphs: WorkloadMix . . . . . . . . . . . . . . . . . . . . . Performance PDFs: WorkloadMix . . . . . . . . . . . . . . . . . . . . . Bandwidth relative fitness PDFs: WorkloadMix . . . . . . . . . . . . . . Throughput relative fitness PDFs: WorkloadMix . . . . . . . . . . . . . Latency relative fitness PDFs: WorkloadMix . . . . . . . . . . . . . . . Workload characteristic CDFs: WorkloadMix . . . . . . . . . . . . . . . Error vs. training: WorkloadMix model training (Median relative error) Error vs. training: WorkloadMix model training (Mean relative error) .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
131 132 133 134 135 136 139 140
K.1 K.2 K.3 K.4 K.5 K.6
Overall relative error CDFs: FitnessDirect model testing on FitnessDirect samples . . . Bandwidth relative error CDFs: FitnessDirect model testing on FitnessDirect samples . Throughput relative error CDFs: FitnessDirect model testing on FitnessDirect samples Latency relative error CDFs: FitnessDirect model testing on FitnessDirect samples . . . Relative error PDFs: FitnessDirect model testing on FitnessDirect samples . . . . . . . Error reduction PDFs: FitnessDirect model testing on FitnessDirect samples . . . . . .
. . . . . .
. . . . . .
. . . . . .
142 143 144 145 146 147
. . . . . . . .
7
LIST OF FIGURES L.1 L.2 L.3 L.4 L.5 L.6
Overall relative error CDFs: FitnessBuffered model testing on FitnessBuffered samples . . . Bandwidth relative error CDFs: FitnessBuffered model testing on FitnessBuffered samples . Throughput relative error CDFs: FitnessBuffered model testing on FitnessBuffered samples Latency relative error CDFs: FitnessBuffered model testing on FitnessBuffered samples . . Relative error PDFs: FitnessBuffered model testing on FitnessBuffered samples . . . . . . . Error reduction PDFs: FitnessBuffered model testing on FitnessBuffered samples . . . . . .
. . . . . .
152 153 154 155 156 157
M.1 M.2 M.3 M.4 M.5 M.6
Overall relative error CDFs: FitnessFS model testing on FitnessFS samples . . . Bandwidth relative error CDFs: FitnessFS model testing on FitnessFS samples . Throughput relative error CDFs: FitnessFS model testing on FitnessFS samples Latency relative error CDFs: FitnessFS model testing on FitnessFS samples . . . Relative error PDFs: FitnessFS model testing on FitnessFS samples . . . . . . . Error reduction PDFs: FitnessFS model testing on FitnessFS samples . . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
162 163 164 165 166 167
N.1 N.2 N.3 N.4 N.5 N.6
Overall relative error CDFs: FitnessCache model testing on FitnessCache samples . . Bandwidth relative error CDFs: FitnessCache model testing on FitnessCache samples Throughput relative error CDFs: FitnessCache model testing on FitnessCache samples Latency relative error CDFs: FitnessCache model testing on FitnessCache samples . . Relative error PDFs: FitnessCache model testing on FitnessCache samples . . . . . . Error reduction PDFs: FitnessCache model testing on FitnessCache samples . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
172 173 174 175 176 177
O.1 O.2 O.3 O.4 O.5 O.6
Overall relative error CDFs: Postmark model testing on Postmark samples . . Bandwidth relative error CDFs: Postmark model testing on Postmark samples Throughput relative error CDFs: Postmark model testing on Postmark samples Latency relative error CDFs: Postmark model testing on Postmark samples . . Relative error PDFs: Postmark model testing on Postmark samples . . . . . . Error reduction PDFs: Postmark model testing on Postmark samples . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
182 183 184 185 186 187
P.1 P.2 P.3 P.4 P.5 P.6
Overall relative error CDFs: Cello model testing on Cello samples . . Bandwidth relative error CDFs: Cello model testing on Cello samples Throughput relative error CDFs: Cello model testing on Cello samples Latency relative error CDFs: Cello model testing on Cello samples . . Relative error PDFs: Cello model testing on Cello samples . . . . . . . Error reduction PDFs: Cello model testing on Cello samples . . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
192 193 194 195 196 197
Q.1 Q.2 Q.3 Q.4 Q.5 Q.6
Overall relative error CDFs: TPC-C model testing on TPC-C samples . . . Bandwidth relative error CDFs: TPC-C model testing on TPC-C samples . Throughput relative error CDFs: TPC-C model testing on TPC-C samples Latency relative error CDFs: TPC-C model testing on TPC-C samples . . . Relative error PDFs: TPC-C model testing on TPC-C samples . . . . . . . Error reduction PDFs: TPC-C model testing on TPC-C samples . . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
202 203 204 205 206 207
R.1 R.2 R.3 R.4 R.5 R.6
Overall relative error CDFs: WorkloadMix model testing on FitnessDirect samples . . . Bandwidth relative error CDFs: WorkloadMix model testing on FitnessDirect samples . Throughput relative error CDFs: WorkloadMix model testing on FitnessDirect samples Latency relative error CDFs: WorkloadMix model testing on FitnessDirect samples . . . Relative error PDFs: WorkloadMix model testing on FitnessDirect samples . . . . . . . Error reduction PDFs: WorkloadMix model testing on FitnessDirect samples . . . . . .
. . . . . .
. . . . . .
. . . . . .
212 213 214 215 216 217
S.1 S.2 S.3 S.4 S.5 S.6
Overall relative error CDFs: WorkloadMix model testing on FitnessBuffered samples . . . Bandwidth relative error CDFs: WorkloadMix model testing on FitnessBuffered samples . Throughput relative error CDFs: WorkloadMix model testing on FitnessBuffered samples Latency relative error CDFs: WorkloadMix model testing on FitnessBuffered samples . . Relative error PDFs: WorkloadMix model testing on FitnessBuffered samples . . . . . . . Error reduction PDFs: WorkloadMix model testing on FitnessBuffered samples . . . . . .
. . . . . .
. . . . . .
222 223 224 225 226 227
. . . . . .
. . . . . .
. . . . . .
. . . . . .
8
LIST OF FIGURES T.1 T.2 T.3 T.4 T.5 T.6
Overall relative error CDFs: WorkloadMix model testing on FitnessFS samples . . . Bandwidth relative error CDFs: WorkloadMix model testing on FitnessFS samples . Throughput relative error CDFs: WorkloadMix model testing on FitnessFS samples Latency relative error CDFs: WorkloadMix model testing on FitnessFS samples . . . Relative error PDFs: WorkloadMix model testing on FitnessFS samples . . . . . . . Error reduction PDFs: WorkloadMix model testing on FitnessFS samples . . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
232 233 234 235 236 237
U.1 U.2 U.3 U.4 U.5 U.6
Overall relative error CDFs: WorkloadMix model testing on FitnessCache samples . . . Bandwidth relative error CDFs: WorkloadMix model testing on FitnessCache samples . Throughput relative error CDFs: WorkloadMix model testing on FitnessCache samples Latency relative error CDFs: WorkloadMix model testing on FitnessCache samples . . . Relative error PDFs: WorkloadMix model testing on FitnessCache samples . . . . . . . Error reduction PDFs: WorkloadMix model testing on FitnessCache samples . . . . . .
. . . . . .
. . . . . .
. . . . . .
242 243 244 245 246 247
V.1 V.2 V.3 V.4 V.5 V.6
Overall relative error CDFs: WorkloadMix model testing on Postmark samples . . . Bandwidth relative error CDFs: WorkloadMix model testing on Postmark samples . Throughput relative error CDFs: WorkloadMix model testing on Postmark samples Latency relative error CDFs: WorkloadMix model testing on Postmark samples . . . Relative error PDFs: WorkloadMix model testing on Postmark samples . . . . . . . Error reduction PDFs: WorkloadMix model testing on Postmark samples . . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
252 253 254 255 256 257
W.1 W.2 W.3 W.4 W.5 W.6
Overall relative error CDFs: WorkloadMix model testing on Cello samples . . . Bandwidth relative error CDFs: WorkloadMix model testing on Cello samples . Throughput relative error CDFs: WorkloadMix model testing on Cello samples Latency relative error CDFs: WorkloadMix model testing on Cello samples . . Relative error PDFs: WorkloadMix model testing on Cello samples . . . . . . . Error reduction PDFs: WorkloadMix model testing on Cello samples . . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
262 263 264 265 266 267
X.1 X.2 X.3 X.4 X.5 X.6
Overall relative error CDFs: WorkloadMix model testing on TPC-C samples . . Bandwidth relative error CDFs: WorkloadMix model testing on TPC-C samples Throughput relative error CDFs: WorkloadMix model testing on TPC-C samples Latency relative error CDFs: WorkloadMix model testing on TPC-C samples . . Relative error PDFs: WorkloadMix model testing on TPC-C samples . . . . . . . Error reduction PDFs: WorkloadMix model testing on TPC-C samples . . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
272 273 274 275 276 277
List of Tables A.1 A.2 A.3 A.4 A.5 A.6 A.7
Predictor rankings: Per-application summary . Tree sizes: Per-application summary . . . . . . Model scoring: Per-application summary . . . . Median relative error: Per-application summary Mean relative error: Per-application summary . Median relative error: Per-application summary Mean relative error: Per-application summary .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
14 15 23 24 25 27 28
B.1 B.2 B.3 B.4 B.5 B.6 B.7
Predictor rankings: Mixed-model summary . Tree sizes: Mixed-model summary . . . . . . Model scoring: Mixed-model summary . . . . Median relative error: Mixed-model summary Mean relative error: Mixed-model summary . Median relative error: Mixed-model summary Mean relative error: Mixed-model summary .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
30 31 39 40 41 43 44
C.1 C.2 C.3 C.4 C.5
Training samples: FitnessDirect . . . . . Workload averages: FitnessDirect . . . . Workload details: FitnessDirect . . . . . Predictor rankings: FitnessDirect (train) Tree sizes: FitnessDirect . . . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
45 45 46 53 54
D.1 D.2 D.3 D.4 D.5
Training samples: FitnessBuffered . . . . . Workload averages: FitnessBuffered . . . . Workload details: FitnessBuffered . . . . . Predictor rankings: FitnessBuffered (train) Tree sizes: FitnessBuffered . . . . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
57 57 58 65 66
E.1 E.2 E.3 E.4 E.5
Training samples: FitnessFS . . . . . Workload averages: FitnessFS . . . . Workload details: FitnessFS . . . . . Predictor rankings: FitnessFS (train) Tree sizes: FitnessFS . . . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
69 69 70 77 78
F.1 F.2 F.3 F.4 F.5
Training samples: FitnessCache . . . . . Workload averages: FitnessCache . . . . Workload details: FitnessCache . . . . . Predictor rankings: FitnessCache (train) Tree sizes: FitnessCache . . . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
81 81 82 89 90
G.1 G.2 G.3 G.4
Training samples: Postmark . . . . . Workload averages: Postmark . . . . Workload details: Postmark . . . . . Predictor rankings: Postmark (train)
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. 93 . 93 . 94 . 101
. . . . .
. . . .
. . . .
. . . . .
9
10
LIST OF TABLES
G.5 Tree sizes: Postmark . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 H.1 H.2 H.3 H.4 H.5
Training samples: Cello . . . . . Workload averages: Cello . . . . Workload details: Cello . . . . . Predictor rankings: Cello (train) Tree sizes: Cello . . . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
105 105 106 113 114
I.1 I.2 I.3 I.4 I.5
Training samples: TPC-C . . . . . Workload averages: TPC-C . . . . Workload details: TPC-C . . . . . Predictor rankings: TPC-C (train) Tree sizes: TPC-C . . . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
117 117 118 125 126
J.1 J.2 J.3 J.4 J.5
Training samples: WorkloadMix . . . . . . Workload averages: WorkloadMix . . . . . Workload details: WorkloadMix . . . . . . Predictor rankings: WorkloadMix (train) Tree sizes: WorkloadMix . . . . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
129 129 130 137 138
K.1 K.2 K.3 K.4
Testing samples: FitnessDirect . . . Model scoring: FitnessDirect . . . . Median relative error: FitnessDirect Mean relative error: FitnessDirect .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
141 148 149 150
L.1 L.2 L.3 L.4
Testing samples: FitnessBuffered . . . Model scoring: FitnessBuffered . . . . Median relative error: FitnessBuffered Mean relative error: FitnessBuffered .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
151 158 159 160
M.1 M.2 M.3 M.4
Testing samples: FitnessFS . . . Model scoring: FitnessFS . . . . Median relative error: FitnessFS Mean relative error: FitnessFS .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
161 168 169 170
N.1 N.2 N.3 N.4
Testing samples: FitnessCache . . . Model scoring: FitnessCache . . . . Median relative error: FitnessCache Mean relative error: FitnessCache .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
171 178 179 180
O.1 O.2 O.3 O.4
Testing samples: Postmark . . . Model scoring: Postmark . . . . Median relative error: Postmark Mean relative error: Postmark .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
181 188 189 190
P.1 P.2 P.3 P.4
Testing samples: Cello . . . Model scoring: Cello . . . . Median relative error: Cello Mean relative error: Cello .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
191 198 199 200
Q.1 Q.2 Q.3 Q.4
Testing samples: TPC-C . . . Model scoring: TPC-C . . . . Median relative error: TPC-C Mean relative error: TPC-C .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
201 208 209 210
. . . .
R.1 Testing samples: FitnessDirect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
11
LIST OF TABLES
R.2 Model scoring: FitnessDirect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 R.3 Median relative error: FitnessDirect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 R.4 Mean relative error: FitnessDirect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 S.1 S.2 S.3 S.4
Testing samples: FitnessBuffered . . . Model scoring: FitnessBuffered . . . . Median relative error: FitnessBuffered Mean relative error: FitnessBuffered .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
221 228 229 230
T.1 T.2 T.3 T.4
Testing samples: FitnessFS . . . Model scoring: FitnessFS . . . . Median relative error: FitnessFS Mean relative error: FitnessFS .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
231 238 239 240
U.1 U.2 U.3 U.4
Testing samples: FitnessCache . . . Model scoring: FitnessCache . . . . Median relative error: FitnessCache Mean relative error: FitnessCache .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
241 248 249 250
V.1 V.2 V.3 V.4
Testing samples: Postmark . . . Model scoring: Postmark . . . . Median relative error: Postmark Mean relative error: Postmark .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
251 258 259 260
W.1 W.2 W.3 W.4
Testing samples: Cello . . . Model scoring: Cello . . . . Median relative error: Cello Mean relative error: Cello .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
261 268 269 270
X.1 X.2 X.3 X.4
Testing samples: TPC-C . . . Model scoring: TPC-C . . . . Median relative error: TPC-C Mean relative error: TPC-C .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
271 278 279 280
. . . .
LIST OF TABLES
12
Appendix A
Per-application summary
13
14
Overall Predictor Read queue Write fraction Read jump Read size Write size Write queue Write jump
Score 1.00 0.72 0.56 0.40 0.37 0.37 0.19
Overall Predictor Read queue Write fraction Read jump Read size Write size Write queue Write jump
Score 1.00 0.65 0.57 0.42 0.39 0.37 0.24
Overall Predictor Latency Bandwidth Throughput Read queue Read jump Write fraction Read size Write size Read latency Write queue Write jump Write latency
Score 1.00 0.92 0.79 0.32 0.28 0.26 0.22 0.21 0.15 0.12 0.10 0.09
Overall Predictor Latency Write fraction Bandwidth Read latency Read jump Read queue Throughput Read size Write size Write queue Write jump Write latency
Score 1.00 0.92 0.89 0.61 0.61 0.60 0.60 0.45 0.33 0.32 0.28 0.25
Absolute Bandwidth Throughput Predictor Score Predictor Score Read queue 1.00 Read queue 1.00 Read jump 0.70 Read jump 0.66 Read size 0.54 Write size 0.50 Write size 0.51 Read size 0.46 Write fraction 0.49 Write queue 0.24 Write queue 0.30 Write jump 0.21 Write jump 0.29 Write fraction 0.19 Relative Bandwidth Throughput Predictor Score Predictor Score Read queue 1.00 Read queue 1.00 Read jump 0.72 Read jump 0.67 Read size 0.52 Write size 0.47 Write fraction 0.52 Read size 0.46 Write size 0.48 Write queue 0.28 Write jump 0.34 Write fraction 0.22 Write queue 0.27 Write jump 0.20 Relative Performance Bandwidth Throughput Predictor Score Predictor Score Bandwidth 1.00 Throughput 1.00 Read jump 0.16 Read queue 0.22 Write fraction 0.14 Read size 0.15 Read queue 0.13 Write size 0.13 Latency 0.11 Read jump 0.13 Write size 0.10 Bandwidth 0.11 Read latency 0.09 Latency 0.11 Read size 0.07 Write fraction 0.07 Throughput 0.07 Write queue 0.05 Write jump 0.06 Write jump 0.03 Write queue 0.04 Read latency 0.02 Write latency 0.02 Write latency 0.02 Relative Fitness Bandwidth Throughput Predictor Score Predictor Score Bandwidth 1.00 Write fraction 1.00 Write fraction 0.99 Bandwidth 0.98 Latency 0.91 Latency 0.92 Read jump 0.85 Read latency 0.85 Read queue 0.82 Read jump 0.77 Read latency 0.74 Throughput 0.64 Throughput 0.66 Read queue 0.60 Read size 0.46 Read size 0.40 Write size 0.41 Write size 0.36 Write queue 0.40 Write queue 0.34 Write latency 0.23 Write latency 0.30 Write jump 0.17 Write jump 0.22
Latency Predictor Write fraction Read queue Write queue Read jump Read size Write size Write jump
Score 1.00 0.72 0.39 0.24 0.16 0.09 0.05
Latency Predictor Write fraction Read queue Write queue Read jump Read size Write size Write jump
Score 1.00 0.88 0.48 0.32 0.28 0.23 0.16
Latency Predictor Latency Write fraction Bandwidth Read queue Read jump Read latency Write latency Read size Write queue Throughput Write size Write jump
Score 1.00 0.13 0.11 0.09 0.08 0.08 0.08 0.08 0.06 0.06 0.05 0.04
Latency Predictor Latency Write fraction Bandwidth Throughput Read size Write jump Read queue Read latency Read jump Write size Write latency Write queue
Score 1.00 0.70 0.64 0.45 0.43 0.37 0.37 0.26 0.24 0.22 0.21 0.21
Table A.1: The normalized importance measure of each predictor.
Model Absolute Relative Relative Performance Relative Fitness
Overall 8.2 8.2 8.3 6.3
Absolute Array Array Array Array Relative Array Array Array Array Relative Performance Array Array Array Array Relative Fitness Array Array Array Array
Absolute Array A Array B Array C Array D
A 10.9 -
Relative Array A Array B Array C Array D
A 10.9 10.7 10.0 12.3
Relative Performance Array A Array B Array C Array D
A 10.7 7.6 8.3 11.1
Relative Fitness Array A Array B Array C Array D
A 1 5.3 11.0 8.3
Bandwidth 9.2 8.6 9.2 6.8
Pairwise Array 9.2 Array 9.2 9.1 9.3 8.6 Array 10.7 8.0 7.9 10.1 Array 1 7.1 8.3 7.6
Bandwidth B C D 10.0 6.1 9.9 Bandwidth B C D 7.9 7.7 8.1 10.0 5.7 8.3 10.1 6.1 6.4 10.6 5.1 9.9 Bandwidth B C D 10.1 8.6 9.7 10.6 6.7 7.9 10.0 11.0 10.3 10.9 9.4 10.4 Bandwidth B C D 6.1 6.0 6.1 1 4.7 5.3 10.0 1 7.3 6.4 4.9 1
Array 8.2 Array 8.2 8.2 10.4 8.2 Array 8.6 10.8 10.0 9.6 Array 6.3 1 8.1 5.3
Throughput 6.4 6.7 7.2 6.5
Array 8.2 Array 7.3 7.2 8.2 6.2 Array 7.0 7.1 10.8 6.6 Array 5.0 5.1 1 4.0
Throughput B C 6.9 8.3 Throughput A B C 7.4 7.4 4.3 7.7 6.9 6.3 8.1 10.7 8.3 4.6 5.9 3.6 Throughput A B C 10.7 8.1 4.9 7.6 10.3 5.7 6.0 10.0 10.9 9.3 8.1 4.3 Throughput A B C 1 5.4 4.6 8.6 1 7.1 7.4 8.7 1 8.9 5.7 3.1 A 7.4 -
Latency 9.1 9.4 8.6 5.5
Array 7.4 Array 8.3 8.0 7.9 7.4 Array 8.3 8.1 8.3 10.0 Array 6.7 5.4 6.2 1
D 3.0
A 9.3 -
D 6.7 7.7 6.9 3.0
A 9.3 9.0 9.7 8.9
D 7.0 7.7 7.3 8.6
A 10.6 9.0 9.4 9.9
D 6.4 5.4 6.9 1
A 1 7.4 6.6 5.6
Table A.2: Tree sizes (leaf nodes) and their averages.
Latency B C 7.9 10.1 Latency B C 9.3 9.9 7.9 9.7 10.3 10.1 8.3 9.9 Latency B C 7.6 7.7 11.6 8.9 10.1 10.6 9.9 6.1 Latency B C 7.4 4.3 1 3.4 5.7 1 3.7 4.0
D 9.3 D 10.0 8.0 10.4 9.3 D 8.3 8.9 7.4 10.9 D 7.6 5.6 4.4 1
16
Bandwidth error (Per-application summary) Median relative error (Avg.)
Median relative error (Avg.)
Overall error (Per-application summary) 0.26 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
0.28 0.26 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08
70
30 AM (i=j) AM
RF
0.2 0.18 0.16 0.14 0.12 0.1 0.08 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
RM RM’
70 RF
Latency error (Per-application summary) Median relative error (Avg.)
Median relative error (Avg.)
Throughput error (Per-application summary) 0.22
40 50 60 Samples used for training (%)
0.28 0.26 0.24 0.22 0.2 0.18 0.16 0.14 0.12 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
Figure A.1: Error vs. training. The training set size varies from 25% of the collected samples to 75%; the remaining samples are used for testing. For each training set size, the Median relative error is calculated for each pairing of devices. They are then averaged to produce the values shown.
17
APPENDIX A. PER-APPLICATION SUMMARY
Bandwidth error (Per-application summary) Mean relative error (Avg.)
Mean relative error (Avg.)
Overall error (Per-application summary) 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
1.2 1 0.8 0.6 0.4 0.2 0
70
30
RF
AM (i=j) AM
0.3 0.25 0.2 0.15 0.1 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
RM RM’
70 RF
Latency error (Per-application summary) Mean relative error (Avg.)
Mean relative error (Avg.)
Throughput error (Per-application summary) 0.35
40 50 60 Samples used for training (%)
0.65 0.6 0.55 0.5 0.45 0.4 0.35 0.3 0.25 0.2 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
Figure A.2: Error vs. training. The training set size varies from 25% of the collected samples to 75%; the remaining samples are used for testing. For each training set size, the Mean relative error is calculated for each pairing of devices. They are then averaged to produce the values shown.
18
Overall (Per-application summary)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Performance Relative Fitness 0.1
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Overall (Per-application summary)
0.9
0.8
Relative error Overall (Per-application summary)
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute Relative 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Overall (Per-application summary)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Overall (Per-application summary)
Relative error
Figure A.3: The cumulative distribution of relative error over all pairwise predictions.
19
APPENDIX A. PER-APPLICATION SUMMARY
Bandwidth (Per-application summary)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
Relative error
Bandwidth (Per-application summary)
Bandwidth (Per-application summary)
Relative error
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Performance Relative Fitness 0.1
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute Relative 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Bandwidth (Per-application summary)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Bandwidth (Per-application summary)
Relative error
Figure A.4: The cumulative distribution of relative error over all pairwise predictions.
20
Throughput (Per-application summary)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
Relative error
Throughput (Per-application summary)
Throughput (Per-application summary)
Relative error
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Performance Relative Fitness 0.1
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute Relative 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Throughput (Per-application summary)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Throughput (Per-application summary)
Relative error
Figure A.5: The cumulative distribution of relative error over all pairwise predictions.
21
APPENDIX A. PER-APPLICATION SUMMARY
Latency (Per-application summary)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Performance Relative Fitness 0.1
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Latency (Per-application summary)
0.9
0.8
Relative error Latency (Per-application summary)
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute Relative 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Latency (Per-application summary)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Latency (Per-application summary)
Relative error
Figure A.6: The cumulative distribution of relative error over all pairwise predictions.
22
Bandwidth
Fraction
Fraction
Overall 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
, -0
80
)
<-1 [-1 .00 .00 ,
[-0
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
.60
-0.
, -0
80
)
Latency
Fraction
Fraction
Throughput 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
Figure A.7: Probability distributions of the difference in the absolute value of the relative error value−measured value (| predictedmeasured |) between the absolute and relative fitness models, over all predictions. The x value axis represent the amount by which relative error is reduced. Negative values indicate an increase in error.
Bandwidth
Bandwidth Throughput Latency RM RM’
RF
RM RM’
AM (i=j) AM
RF
AM (i=j) AM
Figure A.8: Median relative error.
RM RM’
TPC-C
Cello
Postmark
FS
Buffered
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 Direct
TPC-C
Cello
Postmark
FS
Cache RM RM’
RF
Latency Median relative error
AM (i=j) AM
Buffered
Direct
Median relative error
Throughput 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Cache
AM (i=j) AM
TPC-C
0
Cello
Direct
0.05
Postmark
0.1
Cache
0.15
FS
0.2
0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Buffered
Median relative error
Avg. Median relative error
0.25
RF
23
APPENDIX A. PER-APPLICATION SUMMARY
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 67 / 35 / 333 46 / 33 / 356 53 / 39 / 343 55 / 32 / 348 48 / 25 / 362 43 / 33 / 359 52 / 39 / 344 49 / 40 / 346 48 / 41 / 346 48 / 37 / 350 36 / 25 / 374 57 / 31 / 347 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 42 / 21 / 372 34 / 20 / 381 15 / 9 / 411 56 / 34 / 345 37 / 21 / 377 11 / 6 / 418 57 / 28 / 350 54 / 23 / 358 14 / 8 / 413 51 / 30 / 354 33 / 21 / 381 44 / 23 / 368 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 97 / 17 / 321 120 / 34 / 281 93 / 20 / 322 70 / 24 / 341 90 / 40 / 305 51 / 30 / 354 83 / 24 / 328 63 / 30 / 342 62 / 25 / 348 84 / 30 / 321 61 / 27 / 347 94 / 35 / 306 nan% nan% nan%
Bandwidth AM vs. RM RM vs. RM’ 130 / 181 / 124 128 / 214 / 93 149 / 124 / 162 109 / 203 / 123 150 / 122 / 163 130 / 206 / 99 122 / 174 / 139 157 / 193 / 85 149 / 147 / 139 123 / 184 / 128 178 / 122 / 135 98 / 207 / 130 156 / 157 / 122 149 / 207 / 79 153 / 177 / 105 156 / 187 / 92 155 / 130 / 150 147 / 179 / 109 139 / 156 / 140 139 / 208 / 88 156 / 172 / 107 97 / 223 / 115 173 / 144 / 118 148 / 221 / 66 100% 0% 0% 100% 0% 0% Throughput AM vs. RM RM vs. RM’ 150 / 153 / 132 163 / 172 / 100 187 / 162 / 86 153 / 135 / 147 148 / 206 / 81 95 / 123 / 217 160 / 165 / 110 167 / 186 / 82 217 / 169 / 49 166 / 197 / 72 158 / 219 / 58 149 / 210 / 76 135 / 144 / 156 135 / 141 / 159 129 / 212 / 94 160 / 151 / 124 141 / 217 / 77 170 / 162 / 103 167 / 132 / 136 118 / 207 / 110 170 / 178 / 87 140 / 198 / 97 196 / 155 / 84 171 / 166 / 98 100% 0% 0% 100% 0% 0% Latency AM vs. RM RM vs. RM’ 117 / 190 / 128 140 / 164 / 131 151 / 185 / 99 135 / 172 / 128 108 / 177 / 150 105 / 185 / 145 120 / 155 / 160 143 / 200 / 92 132 / 161 / 142 150 / 165 / 120 148 / 152 / 135 102 / 211 / 122 118 / 184 / 133 110 / 156 / 169 148 / 191 / 96 152 / 177 / 106 122 / 162 / 151 159 / 159 / 117 105 / 190 / 140 155 / 198 / 82 154 / 159 / 122 109 / 197 / 129 146 / 196 / 93 156 / 148 / 131 100% 0% 0% 100% 0% 0%
RM’ vs. RF 151 / 246 / 38 170 / 238 / 27 144 / 255 / 36 140 / 263 / 32 179 / 232 / 24 150 / 247 / 38 164 / 235 / 36 156 / 249 / 30 170 / 236 / 29 146 / 254 / 35 148 / 253 / 34 175 / 232 / 28 100% 0% 0%
AM vs. RF 119 / 294 / 22 164 / 248 / 23 147 / 266 / 22 125 / 281 / 29 165 / 248 / 22 120 / 293 / 22 138 / 269 / 28 137 / 267 / 31 169 / 244 / 22 133 / 275 / 27 126 / 288 / 21 162 / 242 / 31 100% 0% 0%
RM’ vs. RF 141 / 256 / 38 144 / 265 / 26 140 / 256 / 39 132 / 258 / 45 153 / 253 / 29 143 / 256 / 36 185 / 218 / 32 179 / 224 / 32 162 / 245 / 28 165 / 233 / 37 147 / 249 / 39 186 / 224 / 25 100% 0% 0%
AM vs. RF 138 / 272 / 25 160 / 253 / 22 118 / 298 / 19 136 / 261 / 38 162 / 250 / 23 98 / 312 / 25 175 / 241 / 19 173 / 235 / 27 134 / 282 / 19 141 / 261 / 33 135 / 271 / 29 192 / 220 / 23 100% 0% 0%
RM’ vs. RF 185 / 223 / 27 148 / 262 / 25 198 / 213 / 24 176 / 229 / 30 194 / 223 / 18 164 / 237 / 34 178 / 229 / 28 216 / 200 / 19 187 / 227 / 21 164 / 234 / 37 160 / 242 / 33 183 / 226 / 26 100% 0% 0%
AM vs. RF 146 / 269 / 20 130 / 288 / 17 147 / 271 / 17 140 / 273 / 22 171 / 245 / 19 125 / 288 / 22 139 / 278 / 18 200 / 220 / 15 191 / 226 / 18 128 / 289 / 18 137 / 276 / 22 180 / 241 / 14 100% 0% 0%
Table A.3: Models are compared across all pairwise predictions. For each prediction, a score of 1 is assigned to the model with the lowest accuracy. If the accuracy of two models is within 0.01, a score of 1 goes to the tie category (3rd number shown). Hypotheses are tested in turn. Specifically, idealized absolute models (i = j) are compared against in-practice absolute models (i 6= j), relative models are compared to absolute models (AM vs. RM), relative models that use performance are compared to those that do not (RM vs. RM’), and relative fitness models are compared to relative models that use performance (RM’ vs. RF). In addition, relative fitness models are compared directly to absolute models (AM vs. RF).
24
Average Metric Bandwidth Throughput Latency
i=j Absolute 0.22 0.17 0.16
Workload Direct Buffered FS Cache Postmark Cello TPC-C
i=j Absolute 0.21 0.17 0.17 0.65 0.07 0.18 0.07
Workload Direct Buffered FS Cache Postmark Cello TPC-C
i=j Absolute 0.23 0.20 0.15 0.36 0.06 0.10 0.07
Workload Direct Buffered FS Cache Postmark Cello TPC-C
i=j Absolute 0.25 0.23 0.15 0.18 0.08 0.11 0.11
Absolute Relative 0.23 0.23 0.17 0.17 0.21 0.18 Bandwidth Absolute Relative 0.24 0.23 0.19 0.18 0.17 0.16 0.67 0.70 0.08 0.07 0.20 0.20 0.07 0.07 Throughput Absolute Relative 0.26 0.22 0.20 0.20 0.15 0.14 0.36 0.36 0.07 0.06 0.10 0.11 0.07 0.07 Latency Absolute 0.34 0.25 0.17 0.20 0.10 0.17 0.23
Relative 0.25 0.23 0.16 0.18 0.09 0.19 0.13
i 6= j Relative’ 0.17 0.15 0.15
Relative fitness 0.11 0.10 0.14
i 6= j Relative’ 0.19 0.14 0.11 0.47 0.07 0.15 0.07
Relative fitness 0.14 0.10 0.09 0.14 0.05 0.13 0.09
i 6= j Relative’ 0.22 0.15 0.11 0.36 0.06 0.09 0.07
Relative fitness 0.14 0.10 0.09 0.13 0.07 0.09 0.08
i 6= j Relative’ 0.20 0.16 0.14 0.17 0.10 0.17 0.13
Relative fitness 0.14 0.13 0.12 0.14 0.09 0.20 0.16
Table A.4: Median relative error.
25
APPENDIX A. PER-APPLICATION SUMMARY
Average Metric Bandwidth Throughput Latency
i=j Absolute 0.57 0.25 0.28
Workload Direct Buffered FS Cache Postmark Cello TPC-C
i=j Absolute 0.62 0.34 0.29 2.11 0.33 0.21 0.09
Workload Direct Buffered FS Cache Postmark Cello TPC-C
i=j Absolute 0.35 0.29 0.23 0.50 0.12 0.14 0.10
Workload Direct Buffered FS Cache Postmark Cello TPC-C
i=j Absolute 0.45 0.35 0.22 0.38 0.23 0.20 0.15
Absolute Relative 0.59 0.61 0.26 0.24 0.43 0.33 Bandwidth Absolute Relative 0.72 0.56 0.34 0.32 0.31 0.29 2.10 2.43 0.34 0.31 0.23 0.25 0.10 0.10 Throughput Absolute Relative 0.37 0.35 0.30 0.27 0.23 0.20 0.52 0.49 0.14 0.12 0.15 0.14 0.10 0.10 Latency Absolute 1.06 0.39 0.25 0.46 0.25 0.33 0.29
Relative 0.47 0.37 0.25 0.43 0.24 0.34 0.18
i 6= j Relative’ 0.39 0.26 0.30
Relative fitness 0.17 0.16 0.24
i 6= j Relative’ 0.36 0.21 0.18 1.37 0.32 0.20 0.09
Relative fitness 0.24 0.16 0.13 0.26 0.11 0.18 0.11
i 6= j Relative’ 0.35 0.23 0.16 0.71 0.13 0.13 0.09
Relative fitness 0.24 0.17 0.13 0.25 0.13 0.12 0.11
i 6= j Relative’ 0.34 0.26 0.22 0.55 0.22 0.33 0.19
Relative fitness 0.25 0.21 0.21 0.29 0.16 0.33 0.26
Table A.5: Mean relative error.
26
Bandwidth
0.6
Mean relative error
0.5 0.4 0.3 0.2
2.5 2 1.5 1 0.5
RF
Throughput
RF
Latency Mean relative error
0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1.2 1 0.8 0.6 0.4 0.2
AM (i=j) AM
RM RM’
RF
AM (i=j) AM
Figure A.9: Mean relative error.
RM RM’
TPC-C
Cello
Postmark
Cache
FS
Buffered
Direct
TPC-C
Cello
Postmark
Cache
FS
Buffered
0 Direct
Mean relative error
Cello
FS
RM RM’
AM (i=j) AM
TPC-C
Latency
RM RM’
Postmark
Bandwidth Throughput AM (i=j) AM
Cache
0
Buffered
0
0.1
Direct
Avg. Mean relative error
0.7
RF
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.17 0.18 0.17 0.14 0.11
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.17 0.18 0.18 0.17
Relative Array A Array B Array C Array D
A 0.17 0.15 0.17 0.16
Relative Performance Array A Array B Array C Array D
A 0.04 0.12 0.14 0.12
Relative Fitness Array A Array B Array C Array D
A 0.0 0.08 0.10 0.09
Bandwidth 0.17 0.18 0.17 0.13 0.10
Pairwise Array A Array 0.15 0.18 0.18 0.14 0.18 0.16 0.18 0.16 Array A Array 0.15 0.15 0.16 0.14 0.16 0.15 0.16 0.16 Array A Array 0.04 0.12 0.13 0.04 0.14 0.14 0.13 0.11 Array A Array 0.0 0.09 0.09 0.0 0.11 0.12 0.10 0.08
Bandwidth B C 0.19 0.17 0.16 0.18 0.17 0.16 0.17 0.18 Bandwidth B C 0.17 0.18 0.16 0.18 0.16 0.16 0.16 0.20 Bandwidth B C 0.12 0.13 0.04 0.14 0.15 0.03 0.11 0.14 Bandwidth B C 0.08 0.12 0.0 0.13 0.10 0.0 0.08 0.12
D 0.18 0.17 0.18 0.17
A 0.13 0.14 0.14 0.14
D 0.19 0.19 0.18 0.17
A 0.13 0.14 0.13 0.15
D 0.14 0.12 0.16 0.04
A 0.03 0.11 0.13 0.12
D 0.10 0.09 0.12 0.0
A 0.0 0.08 0.11 0.09
B
B
B
B
Throughput 0.16 0.17 0.15 0.13 0.10
Array 0.19 0.20 0.17 0.19 Array 0.19 0.20 0.17 0.19 Array 0.17 0.17 0.04 0.17 Array 0.12 0.14 0.0 0.14
Throughput B C 0.13 0.16 0.12 0.16 0.14 0.15 0.13 0.16 Throughput B C 0.13 0.17 0.12 0.19 0.11 0.15 0.14 0.17 Throughput B C 0.11 0.17 0.03 0.18 0.12 0.03 0.10 0.16 Throughput B C 0.08 0.11 0.0 0.12 0.11 0.0 0.07 0.14
Table A.6: Median relative error
C
C
C
C
Latency 0.17 0.21 0.18 0.16 0.13
Array 0.21 0.20 0.21 0.20 Array 0.18 0.18 0.17 0.20 Array 0.14 0.13 0.17 0.04 Array 0.11 0.09 0.13 0.0
D 0.25 0.25 0.25 0.24
A 0.17 0.21 0.22 0.22
D 0.18 0.18 0.19 0.24
A 0.17 0.19 0.19 0.16
D 0.16 0.12 0.18 0.05
A 0.04 0.16 0.16 0.14
D 0.10 0.08 0.12 0.0
A 0.0 0.13 0.14 0.12
D
D
D
D
Latency B C 0.20 0.25 0.15 0.25 0.18 0.20 0.18 0.24 Latency B C 0.16 0.23 0.15 0.22 0.17 0.20 0.18 0.20 Latency B C 0.14 0.20 0.04 0.20 0.16 0.05 0.12 0.22 Latency B C 0.13 0.12 0.0 0.17 0.16 0.0 0.10 0.18
D 0.21 0.18 0.19 0.17 D 0.17 0.17 0.17 0.17 D 0.12 0.13 0.17 0.04 D 0.12 0.10 0.15 0.0
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.36 0.43 0.37 0.29 0.19
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.38 0.47 0.46 0.45
Relative Array A Array B Array C Array D
A 0.38 0.39 0.40 0.40
Relative Performance Array A Array B Array C Array D
A 0.14 0.33 0.28 0.29
Relative Fitness Array A Array B Array C Array D
A 0.0 0.16 0.16 0.13
Bandwidth 0.50 0.53 0.51 0.33 0.17
Pairwise Array A Array 0.28 0.58 0.32 0.38 0.33 0.39 0.32 0.38 Array A Array 0.28 0.41 0.30 0.38 0.30 0.40 0.30 0.35 Array A Array 0.09 0.24 0.25 0.10 0.24 0.27 0.23 0.23 Array A Array 0.0 0.16 0.17 0.0 0.17 0.22 0.15 0.14
Bandwidth B C 0.62 0.50 0.59 0.63 0.55 0.53 0.54 0.57 Bandwidth B C 0.69 0.51 0.59 0.51 0.66 0.53 0.46 0.50 Bandwidth B C 0.29 0.47 0.13 0.48 0.31 0.15 0.28 0.33 Bandwidth B C 0.15 0.19 0.0 0.24 0.20 0.0 0.13 0.21
D 0.55 0.54 0.52 0.51
A 0.21 0.22 0.22 0.22
D 0.58 0.54 0.51 0.51
A 0.21 0.22 0.23 0.24
D 0.37 0.26 0.29 0.15
A 0.06 0.19 0.20 0.17
D 0.17 0.14 0.20 0.0
A 0.0 0.15 0.17 0.13
B
B
B
B
Throughput 0.26 0.27 0.25 0.25 0.17
Array 0.52 0.44 0.38 0.41 Array 0.39 0.40 0.38 0.38 Array 0.41 0.38 0.12 0.37 Array 0.21 0.26 0.0 0.25
Throughput B C 0.23 0.30 0.21 0.30 0.24 0.30 0.23 0.30 Throughput B C 0.21 0.27 0.21 0.32 0.20 0.30 0.23 0.28 Throughput B C 0.21 0.30 0.07 0.30 0.23 0.06 0.17 0.42 Throughput B C 0.13 0.18 0.0 0.22 0.20 0.0 0.11 0.22
Table A.7: Mean relative error
C
C
C
C
Latency 0.31 0.49 0.34 0.29 0.23
Array 0.66 0.41 0.41 0.40 Array 0.41 0.42 0.38 0.40 Array 0.31 0.27 0.30 0.13 Array 0.19 0.16 0.23 0.0
D 0.34 0.35 0.35 0.34
A 0.25 0.28 0.29 0.29
D 0.28 0.30 0.28 0.34
A 0.25 0.29 0.27 0.25
D 0.27 0.29 0.26 0.13
A 0.08 0.23 0.24 0.22
D 0.18 0.15 0.21 0.0
A 0.0 0.21 0.20 0.19
D
D
D
D
Latency B C 0.88 0.75 0.34 0.39 0.39 0.32 0.37 0.38 Latency B C 0.32 0.41 0.34 0.38 0.34 0.32 0.36 0.35 Latency B C 0.24 0.45 0.09 0.36 0.28 0.14 0.23 0.37 Latency B C 0.21 0.24 0.0 0.32 0.26 0.0 0.17 0.32
D 1.09 0.36 0.36 0.33 D 0.39 0.41 0.36 0.33 D 0.28 0.26 0.33 0.12 D 0.23 0.18 0.28 0.0
Appendix B
Mixed-model summary
29
30
Overall Predictor Read queue Write fraction Read jump Read size Write size Write queue Write jump
Score 1.00 0.72 0.56 0.40 0.37 0.37 0.19
Overall Predictor Read queue Write fraction Read jump Read size Write size Write queue Write jump
Score 1.00 0.65 0.57 0.42 0.39 0.37 0.24
Overall Predictor Latency Bandwidth Throughput Read queue Read jump Write fraction Read size Write size Read latency Write queue Write jump Write latency
Score 1.00 0.92 0.79 0.32 0.28 0.26 0.22 0.21 0.15 0.12 0.10 0.09
Overall Predictor Latency Write fraction Bandwidth Read latency Read jump Read queue Throughput Read size Write size Write queue Write jump Write latency
Score 1.00 0.92 0.89 0.61 0.61 0.60 0.60 0.45 0.33 0.32 0.28 0.25
Absolute Bandwidth Throughput Predictor Score Predictor Score Read queue 1.00 Read queue 1.00 Read jump 0.70 Read jump 0.66 Read size 0.54 Write size 0.50 Write size 0.51 Read size 0.46 Write fraction 0.49 Write queue 0.24 Write queue 0.30 Write jump 0.21 Write jump 0.29 Write fraction 0.19 Relative Bandwidth Throughput Predictor Score Predictor Score Read queue 1.00 Read queue 1.00 Read jump 0.72 Read jump 0.67 Read size 0.52 Write size 0.47 Write fraction 0.52 Read size 0.46 Write size 0.48 Write queue 0.28 Write jump 0.34 Write fraction 0.22 Write queue 0.27 Write jump 0.20 Relative Performance Bandwidth Throughput Predictor Score Predictor Score Bandwidth 1.00 Throughput 1.00 Read jump 0.16 Read queue 0.22 Write fraction 0.14 Read size 0.15 Read queue 0.13 Write size 0.13 Latency 0.11 Read jump 0.13 Write size 0.10 Bandwidth 0.11 Read latency 0.09 Latency 0.11 Read size 0.07 Write fraction 0.07 Throughput 0.07 Write queue 0.05 Write jump 0.06 Write jump 0.03 Write queue 0.04 Read latency 0.02 Write latency 0.02 Write latency 0.02 Relative Fitness Bandwidth Throughput Predictor Score Predictor Score Bandwidth 1.00 Write fraction 1.00 Write fraction 0.99 Bandwidth 0.98 Latency 0.91 Latency 0.92 Read jump 0.85 Read latency 0.85 Read queue 0.82 Read jump 0.77 Read latency 0.74 Throughput 0.64 Throughput 0.66 Read queue 0.60 Read size 0.46 Read size 0.40 Write size 0.41 Write size 0.36 Write queue 0.40 Write queue 0.34 Write latency 0.23 Write latency 0.30 Write jump 0.17 Write jump 0.22
Latency Predictor Write fraction Read queue Write queue Read jump Read size Write size Write jump
Score 1.00 0.72 0.39 0.24 0.16 0.09 0.05
Latency Predictor Write fraction Read queue Write queue Read jump Read size Write size Write jump
Score 1.00 0.88 0.48 0.32 0.28 0.23 0.16
Latency Predictor Latency Write fraction Bandwidth Read queue Read jump Read latency Write latency Read size Write queue Throughput Write size Write jump
Score 1.00 0.13 0.11 0.09 0.08 0.08 0.08 0.08 0.06 0.06 0.05 0.04
Latency Predictor Latency Write fraction Bandwidth Throughput Read size Write jump Read queue Read latency Read jump Write size Write latency Write queue
Score 1.00 0.70 0.64 0.45 0.43 0.37 0.37 0.26 0.24 0.22 0.21 0.21
Table B.1: The normalized importance measure of each predictor.
Model Absolute Relative Relative Performance Relative Fitness
Overall 8.2 8.2 8.3 6.3
Absolute Array Array Array Array Relative Array Array Array Array Relative Performance Array Array Array Array Relative Fitness Array Array Array Array
Absolute Array A Array B Array C Array D
A 10.9 -
Relative Array A Array B Array C Array D
A 10.9 10.7 10.0 12.3
Relative Performance Array A Array B Array C Array D
A 10.7 7.6 8.3 11.1
Relative Fitness Array A Array B Array C Array D
A 1 5.3 11.0 8.3
Bandwidth 9.2 8.6 9.2 6.8
Pairwise Array 9.2 Array 9.2 9.1 9.3 8.6 Array 10.7 8.0 7.9 10.1 Array 1 7.1 8.3 7.6
Bandwidth B C D 10.0 6.1 9.9 Bandwidth B C D 7.9 7.7 8.1 10.0 5.7 8.3 10.1 6.1 6.4 10.6 5.1 9.9 Bandwidth B C D 10.1 8.6 9.7 10.6 6.7 7.9 10.0 11.0 10.3 10.9 9.4 10.4 Bandwidth B C D 6.1 6.0 6.1 1 4.7 5.3 10.0 1 7.3 6.4 4.9 1
Array 8.2 Array 8.2 8.2 10.4 8.2 Array 8.6 10.8 10.0 9.6 Array 6.3 1 8.1 5.3
Throughput 6.4 6.7 7.2 6.5
Array 8.2 Array 7.3 7.2 8.2 6.2 Array 7.0 7.1 10.8 6.6 Array 5.0 5.1 1 4.0
Throughput B C 6.9 8.3 Throughput A B C 7.4 7.4 4.3 7.7 6.9 6.3 8.1 10.7 8.3 4.6 5.9 3.6 Throughput A B C 10.7 8.1 4.9 7.6 10.3 5.7 6.0 10.0 10.9 9.3 8.1 4.3 Throughput A B C 1 5.4 4.6 8.6 1 7.1 7.4 8.7 1 8.9 5.7 3.1 A 7.4 -
Latency 9.1 9.4 8.6 5.5
Array 7.4 Array 8.3 8.0 7.9 7.4 Array 8.3 8.1 8.3 10.0 Array 6.7 5.4 6.2 1
D 3.0
A 9.3 -
D 6.7 7.7 6.9 3.0
A 9.3 9.0 9.7 8.9
D 7.0 7.7 7.3 8.6
A 10.6 9.0 9.4 9.9
D 6.4 5.4 6.9 1
A 1 7.4 6.6 5.6
Table B.2: Tree sizes (leaf nodes) and their averages.
Latency B C 7.9 10.1 Latency B C 9.3 9.9 7.9 9.7 10.3 10.1 8.3 9.9 Latency B C 7.6 7.7 11.6 8.9 10.1 10.6 9.9 6.1 Latency B C 7.4 4.3 1 3.4 5.7 1 3.7 4.0
D 9.3 D 10.0 8.0 10.4 9.3 D 8.3 8.9 7.4 10.9 D 7.6 5.6 4.4 1
32
Bandwidth error (Mixed-model summary) Median relative error (Avg.)
Median relative error (Avg.)
Overall error (Mixed-model summary) 0.26 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
0.28 0.26 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08
70
30 AM (i=j) AM
RF
0.2 0.18 0.16 0.14 0.12 0.1 0.08 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
RM RM’
70 RF
Latency error (Mixed-model summary) Median relative error (Avg.)
Median relative error (Avg.)
Throughput error (Mixed-model summary) 0.22
40 50 60 Samples used for training (%)
0.28 0.26 0.24 0.22 0.2 0.18 0.16 0.14 0.12 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
Figure B.1: Error vs. training. The training set size varies from 25% of the collected samples to 75%; the remaining samples are used for testing. For each training set size, the Median relative error is calculated for each pairing of devices. They are then averaged to produce the values shown.
33
APPENDIX B. MIXED-MODEL SUMMARY
Bandwidth error (Mixed-model summary) Mean relative error (Avg.)
Mean relative error (Avg.)
Overall error (Mixed-model summary) 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
1.2 1 0.8 0.6 0.4 0.2 0
70
30
RF
0.2 0.15 0.1 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
70 RF
Latency error (Mixed-model summary) Mean relative error (Avg.)
Mean relative error (Avg.)
0.3 0.25
RM RM’
AM (i=j) AM
Throughput error (Mixed-model summary) 0.35
40 50 60 Samples used for training (%)
0.65 0.6 0.55 0.5 0.45 0.4 0.35 0.3 0.25 0.2 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
Figure B.2: Error vs. training. The training set size varies from 25% of the collected samples to 75%; the remaining samples are used for testing. For each training set size, the Mean relative error is calculated for each pairing of devices. They are then averaged to produce the values shown.
34
Overall (Mixed-model summary)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Performance Relative Fitness 0.1
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Overall (Mixed-model summary)
0.9
0.8
Relative error Overall (Mixed-model summary)
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute Relative 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Overall (Mixed-model summary)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Overall (Mixed-model summary)
Relative error
Figure B.3: The cumulative distribution of relative error over all pairwise predictions.
35
APPENDIX B. MIXED-MODEL SUMMARY
Bandwidth (Mixed-model summary)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Performance Relative Fitness 0.1
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Bandwidth (Mixed-model summary)
0.9
0.8
Relative error Bandwidth (Mixed-model summary)
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute Relative 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Bandwidth (Mixed-model summary)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Bandwidth (Mixed-model summary)
Relative error
Figure B.4: The cumulative distribution of relative error over all pairwise predictions.
36
Throughput (Mixed-model summary)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Performance Relative Fitness 0.1
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Throughput (Mixed-model summary)
0.9
0.8
Relative error Throughput (Mixed-model summary)
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute Relative 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Throughput (Mixed-model summary)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Throughput (Mixed-model summary)
Relative error
Figure B.5: The cumulative distribution of relative error over all pairwise predictions.
37
APPENDIX B. MIXED-MODEL SUMMARY
Latency (Mixed-model summary)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Performance Relative Fitness 0.1
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Latency (Mixed-model summary)
0.9
0.8
Relative error Latency (Mixed-model summary)
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute Relative 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Latency (Mixed-model summary)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Latency (Mixed-model summary)
Relative error
Figure B.6: The cumulative distribution of relative error over all pairwise predictions.
38
Bandwidth
Fraction
Fraction
Overall 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
, -0
80
)
<-1 [-1 .00 .00 ,
[-0
)
Latency
Fraction
Fraction
, -0
80
Throughput 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
.60
-0.
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
Figure B.7: Probability distributions of the difference in the absolute value of the relative error value−measured value (| predictedmeasured |) between the absolute and relative fitness models, over all predictions. The x value axis represent the amount by which relative error is reduced. Negative values indicate an increase in error.
Bandwidth
RM RM’
RF
RM RM’
AM (i=j) AM
RM RM’
RF
AM (i=j) AM
Figure B.8: Median relative error.
RM RM’
TPC-C
Cello
Postmark
Cache
FS
Buffered
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 Direct
TPC-C
Cello
Postmark
Cache
FS
Buffered
Median relative error
AM (i=j) AM
RF
Latency
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 Direct
Median relative error
Throughput
TPC-C
Bandwidth Throughput Latency AM (i=j) AM
Cello
0
Postmark
Direct
0.05
Cache
0.1
FS
0.15
0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 Buffered
Median relative error
Avg. Median relative error
0.2
RF
39
APPENDIX B. MIXED-MODEL SUMMARY
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 56 / 38 / 341 87 / 45 / 303 85 / 49 / 301 44 / 32 / 359 64 / 33 / 338 61 / 46 / 328 51 / 33 / 351 54 / 42 / 339 78 / 55 / 302 47 / 38 / 350 52 / 28 / 355 62 / 31 / 342 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 73 / 58 / 304 61 / 45 / 329 71 / 42 / 322 119 / 37 / 279 53 / 31 / 351 43 / 37 / 355 141 / 47 / 247 80 / 50 / 305 63 / 51 / 321 120 / 41 / 274 56 / 42 / 337 51 / 37 / 347 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 129 / 49 / 257 139 / 44 / 252 65 / 30 / 340 96 / 45 / 294 87 / 38 / 310 54 / 26 / 355 99 / 35 / 301 93 / 46 / 296 61 / 31 / 343 95 / 41 / 299 88 / 38 / 309 91 / 44 / 300 nan% nan% nan%
Bandwidth AM vs. RM RM vs. RM’ 162 / 138 / 135 153 / 207 / 75 179 / 202 / 54 180 / 220 / 35 181 / 163 / 91 124 / 230 / 81 156 / 174 / 105 169 / 227 / 39 183 / 175 / 77 139 / 236 / 60 159 / 184 / 92 141 / 219 / 75 146 / 174 / 115 181 / 213 / 41 194 / 166 / 75 165 / 210 / 60 146 / 172 / 117 148 / 229 / 58 156 / 158 / 121 118 / 258 / 59 174 / 152 / 109 141 / 247 / 47 169 / 188 / 78 157 / 225 / 53 100% 0% 0% 100% 0% 0% Throughput AM vs. RM RM vs. RM’ 162 / 172 / 101 159 / 227 / 49 153 / 197 / 85 316 / 115 / 4 165 / 208 / 62 164 / 203 / 68 192 / 209 / 34 151 / 235 / 49 285 / 115 / 35 125 / 288 / 22 148 / 212 / 75 191 / 209 / 35 179 / 218 / 38 144 / 215 / 76 240 / 176 / 19 163 / 248 / 24 176 / 203 / 56 185 / 216 / 34 161 / 213 / 61 138 / 230 / 67 192 / 167 / 76 152 / 223 / 60 172 / 170 / 93 177 / 227 / 31 100% 0% 0% 100% 0% 0% Latency AM vs. RM RM vs. RM’ 173 / 220 / 42 175 / 240 / 20 148 / 213 / 74 145 / 237 / 53 156 / 196 / 83 152 / 219 / 64 161 / 180 / 94 167 / 220 / 48 233 / 159 / 43 169 / 255 / 11 152 / 192 / 91 142 / 245 / 48 145 / 166 / 124 163 / 203 / 69 169 / 178 / 88 179 / 195 / 61 139 / 158 / 138 180 / 175 / 80 158 / 170 / 107 157 / 236 / 42 168 / 201 / 66 162 / 240 / 33 180 / 177 / 78 178 / 207 / 50 100% 0% 0% 100% 0% 0%
RM’ vs. RF 157 / 242 / 36 169 / 245 / 21 175 / 232 / 28 138 / 257 / 40 194 / 214 / 27 160 / 234 / 41 141 / 270 / 24 188 / 214 / 33 186 / 214 / 35 149 / 247 / 39 166 / 227 / 42 184 / 223 / 28 100% 0% 0%
AM vs. RF 150 / 263 / 22 153 / 256 / 26 135 / 283 / 17 116 / 295 / 24 157 / 255 / 23 132 / 277 / 26 131 / 282 / 22 183 / 228 / 24 157 / 248 / 30 126 / 287 / 22 126 / 281 / 28 159 / 243 / 33 100% 0% 0%
RM’ vs. RF 157 / 232 / 46 106 / 317 / 12 158 / 243 / 34 202 / 202 / 31 185 / 225 / 25 171 / 241 / 23 172 / 225 / 38 194 / 205 / 36 174 / 231 / 30 168 / 231 / 36 169 / 233 / 33 213 / 197 / 25 100% 0% 0%
AM vs. RF 143 / 264 / 28 169 / 246 / 20 136 / 283 / 16 144 / 271 / 20 166 / 242 / 27 134 / 274 / 27 137 / 279 / 19 184 / 221 / 30 158 / 245 / 32 124 / 289 / 22 149 / 256 / 30 200 / 210 / 25 100% 0% 0%
RM’ vs. RF 163 / 247 / 25 183 / 234 / 18 192 / 217 / 26 205 / 209 / 21 175 / 240 / 20 172 / 226 / 37 191 / 220 / 24 196 / 213 / 26 179 / 230 / 26 194 / 218 / 23 171 / 235 / 29 181 / 237 / 17 100% 0% 0%
AM vs. RF 122 / 290 / 23 136 / 284 / 15 148 / 265 / 22 169 / 245 / 21 170 / 243 / 22 124 / 288 / 23 161 / 250 / 24 155 / 261 / 19 164 / 251 / 20 140 / 273 / 22 135 / 280 / 20 149 / 264 / 22 100% 0% 0%
Table B.3: Models are compared across all pairwise predictions. For each prediction, a score of 1 is assigned to the model with the lowest accuracy. If the accuracy of two models is within 0.01, a score of 1 goes to the tie category (3rd number shown). Hypotheses are tested in turn. Specifically, idealized absolute models (i = j) are compared against in-practice absolute models (i 6= j), relative models are compared to absolute models (AM vs. RM), relative models that use performance are compared to those that do not (RM vs. RM’), and relative fitness models are compared to relative models that use performance (RM’ vs. RF). In addition, relative fitness models are compared directly to absolute models (AM vs. RF).
40
Average Metric Bandwidth Throughput Latency
i=j Absolute 0.19 0.16 0.16
Workload Direct Buffered FS Cache Postmark Cello TPC-C
i=j Absolute 0.21 0.16 0.16 0.46 0.07 0.17 0.09
Workload Direct Buffered FS Cache Postmark Cello TPC-C
i=j Absolute 0.19 0.19 0.14 0.31 0.06 0.12 0.08
Workload Direct Buffered FS Cache Postmark Cello TPC-C
i=j Absolute 0.29 0.20 0.14 0.16 0.08 0.17 0.10
Absolute Relative 0.20 0.20 0.18 0.16 0.20 0.19 Bandwidth Absolute Relative 0.23 0.21 0.17 0.16 0.17 0.18 0.48 0.45 0.08 0.09 0.18 0.20 0.10 0.08 Throughput Absolute Relative 0.21 0.23 0.19 0.18 0.16 0.15 0.33 0.28 0.08 0.07 0.13 0.16 0.13 0.08 Latency Absolute 0.35 0.21 0.17 0.16 0.13 0.24 0.13
Relative 0.30 0.22 0.17 0.19 0.10 0.20 0.14
i 6= j Relative’ 0.16 0.13 0.16
Relative fitness 0.10 0.10 0.14
i 6= j Relative’ 0.17 0.12 0.12 0.41 0.07 0.17 0.09
Relative fitness 0.13 0.11 0.08 0.12 0.06 0.14 0.09
i 6= j Relative’ 0.20 0.14 0.11 0.19 0.06 0.13 0.08
Relative fitness 0.14 0.10 0.09 0.14 0.07 0.09 0.08
i 6= j Relative’ 0.21 0.15 0.13 0.19 0.10 0.20 0.14
Relative fitness 0.16 0.13 0.10 0.12 0.10 0.22 0.17
Table B.4: Median relative error.
41
APPENDIX B. MIXED-MODEL SUMMARY
Average Metric Bandwidth Throughput Latency
i=j Absolute 0.34 0.26 0.32
Workload Direct Buffered FS Cache Postmark Cello TPC-C
i=j Absolute 0.40 0.24 0.27 1.02 0.15 0.22 0.11
Workload Direct Buffered FS Cache Postmark Cello TPC-C
i=j Absolute 0.26 0.26 0.27 0.44 0.16 0.17 0.26
Workload Direct Buffered FS Cache Postmark Cello TPC-C
i=j Absolute 0.52 0.33 0.30 0.42 0.29 0.25 0.13
Absolute Relative 0.37 0.41 0.29 0.27 0.55 0.43 Bandwidth Absolute Relative 0.42 0.41 0.25 0.23 0.31 0.30 1.02 1.25 0.17 0.23 0.25 0.29 0.16 0.13 Throughput Absolute Relative 0.28 0.40 0.27 0.30 0.30 0.29 0.44 0.39 0.15 0.13 0.19 0.24 0.37 0.15 Latency Absolute 0.63 0.35 0.48 0.80 0.66 0.54 0.42
Relative 0.50 0.33 0.36 0.88 0.26 0.39 0.26
i 6= j Relative’ 0.35 0.43 0.32
Relative fitness 0.34 0.41 0.30
i 6= j Relative’ 0.36 0.19 0.18 1.21 0.15 0.23 0.11
Relative fitness 0.38 0.17 0.13 1.21 0.14 0.20 0.12
i 6= j Relative’ 0.75 0.22 0.17 1.38 0.16 0.21 0.13
Relative fitness 0.25 0.19 0.15 1.86 0.14 0.14 0.11
i 6= j Relative’ 0.35 0.24 0.22 0.59 0.24 0.34 0.23
Relative fitness 0.30 0.20 0.20 0.31 0.22 0.42 0.46
Table B.5: Mean relative error.
42
Bandwidth
Latency
RM RM’
RF
RM RM’
AM (i=j) AM
RF
AM (i=j) AM
Figure B.9: Mean relative error.
RM RM’
TPC-C
Cello
Postmark
FS
Buffered
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Direct
TPC-C
Cello
Postmark
FS
Cache RM RM’
RF
Latency Mean relative error
AM (i=j) AM
Buffered
Direct
Mean relative error
Throughput 2 1.8 1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 0
Cache
Bandwidth Throughput AM (i=j) AM
TPC-C
0
Cello
Direct
0.1
Postmark
0.2
Cache
0.3
FS
0.4
1.4 1.2 1 0.8 0.6 0.4 0.2 0 Buffered
0.5
Mean relative error
Avg. Mean relative error
0.6
RF
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.16 0.18 0.18 0.14 0.11
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.17 0.17 0.18 0.17
Relative Array A Array B Array C Array D
A 0.17 0.16 0.16 0.19
Relative Performance Array A Array B Array C Array D
A 0.01 0.13 0.14 0.12
Relative Fitness Array A Array B Array C Array D
A 0.0 0.08 0.09 0.08
Bandwidth 0.16 0.18 0.18 0.13 0.10
Pairwise Array A Array 0.15 0.16 0.18 0.14 0.19 0.16 0.19 0.16 Array A Array 0.15 0.15 0.16 0.14 0.17 0.17 0.17 0.15 Array A Array 0.01 0.12 0.12 0.01 0.14 0.15 0.12 0.11 Array A Array 0.0 0.09 0.10 0.0 0.10 0.12 0.10 0.09
Bandwidth B C 0.16 0.20 0.15 0.19 0.15 0.17 0.16 0.18 Bandwidth B C 0.15 0.18 0.15 0.20 0.18 0.17 0.17 0.18 Bandwidth B C 0.12 0.15 0.01 0.15 0.15 0.01 0.10 0.14 Bandwidth B C 0.09 0.13 0.0 0.13 0.12 0.0 0.07 0.13
D 0.20 0.18 0.20 0.16
A 0.13 0.18 0.18 0.18
D 0.19 0.17 0.18 0.16
A 0.13 0.14 0.16 0.15
D 0.11 0.12 0.14 0.01
A 0.01 0.09 0.13 0.10
D 0.10 0.08 0.12 0.0
A 0.0 0.10 0.10 0.09
B
B
B
B
Throughput 0.15 0.17 0.16 0.13 0.10
Array 0.20 0.20 0.17 0.19 Array 0.19 0.26 0.17 0.19 Array 0.21 0.17 0.01 0.15 Array 0.13 0.14 0.0 0.14
Throughput B C 0.14 0.17 0.12 0.18 0.14 0.16 0.13 0.17 Throughput B C 0.12 0.15 0.12 0.30 0.18 0.16 0.12 0.17 Throughput B C 0.10 0.34 0.01 0.15 0.12 0.01 0.10 0.14 Throughput B C 0.07 0.12 0.0 0.12 0.12 0.0 0.08 0.15
Table B.6: Median relative error
C
C
C
C
Latency 0.17 0.21 0.19 0.15 0.13
Array 0.20 0.19 0.20 0.18 Array 0.18 0.17 0.18 0.18 Array 0.13 0.12 0.15 0.01 Array 0.11 0.09 0.12 0.0
D 0.21 0.20 0.20 0.20
A 0.15 0.19 0.20 0.20
D 0.18 0.17 0.19 0.20
A 0.15 0.17 0.18 0.17
D 0.14 0.14 0.14 0.01
A 0.01 0.14 0.15 0.13
D 0.10 0.10 0.11 0.0
A 0.0 0.14 0.12 0.12
D
D
D
D
Latency B C 0.24 0.30 0.17 0.22 0.20 0.18 0.20 0.22 Latency B C 0.18 0.23 0.17 0.30 0.17 0.18 0.17 0.21 Latency B C 0.16 0.19 0.01 0.21 0.16 0.01 0.13 0.19 Latency B C 0.12 0.14 0.0 0.17 0.15 0.0 0.11 0.16
D 0.18 0.19 0.19 0.17 D 0.18 0.18 0.18 0.17 D 0.12 0.11 0.16 0.01 D 0.12 0.09 0.13 0.0
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.31 0.39 0.36 0.33 0.29
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.33 0.37 0.38 0.38
Relative Array A Array B Array C Array D
A 0.33 0.42 0.34 0.44
Relative Performance Array A Array B Array C Array D
A 0.04 0.27 0.29 0.23
Relative Fitness Array A Array B Array C Array D
A 0.0 0.14 0.21 0.13
Bandwidth 0.32 0.35 0.37 0.30 0.28
Pairwise Array A Array 0.27 0.43 0.32 0.29 0.32 0.32 0.32 0.37 Array A Array 0.27 0.32 0.32 0.29 0.29 0.33 0.33 0.34 Array A Array 0.03 0.26 0.24 0.03 0.24 0.43 0.21 0.23 Array A Array 0.0 0.18 0.19 0.0 0.20 0.83 0.15 0.19
Bandwidth B C 0.30 0.41 0.28 0.40 0.29 0.37 0.30 0.39 Bandwidth B C 0.33 0.39 0.28 0.39 0.34 0.37 0.34 0.38 Bandwidth B C 0.27 0.41 0.04 0.42 0.26 0.04 0.23 0.31 Bandwidth B C 0.14 0.21 0.0 0.61 1.09 0.0 0.13 0.22
D 0.33 0.31 0.32 0.31
A 0.22 0.28 0.29 0.27
D 0.36 0.35 0.32 0.31
A 0.22 0.27 0.25 0.25
D 0.30 0.32 0.32 0.07
A 0.01 0.21 0.20 0.16
D 0.17 0.14 0.21 0.0
A 0.0 0.20 0.17 0.14
B
B
B
B
Throughput 0.26 0.28 0.29 0.41 0.31
Array 0.51 0.44 0.32 0.37 Array 0.35 0.58 0.32 0.38 Array 0.42 0.69 0.04 0.29 Array 0.24 0.41 0.0 0.24
Throughput B C 0.20 0.30 0.20 0.31 0.21 0.28 0.22 0.29 Throughput B C 0.20 0.25 0.20 0.59 0.24 0.28 0.26 0.29 Throughput B C 0.17 0.54 0.02 1.13 0.76 0.02 0.15 0.24 Throughput B C 0.13 0.21 0.0 0.20 1.10 0.0 0.14 0.22
Table B.7: Mean relative error
C
C
C
C
Latency 0.35 0.53 0.41 0.30 0.27
Array 0.49 0.37 0.38 0.36 Array 0.35 0.33 0.36 0.36 Array 0.26 0.31 0.42 0.04 Array 0.20 0.18 0.44 0.0
D 0.33 0.32 0.31 0.32
A 0.25 0.30 0.29 0.30
D 0.31 0.26 0.36 0.32
A 0.25 0.26 0.29 0.31
D 0.24 0.43 0.64 0.02
A 0.02 0.24 0.24 0.23
D 0.17 0.19 0.85 0.0
A 0.0 0.24 0.23 0.19
D
D
D
D
Latency B C 0.79 0.82 0.39 0.63 0.44 0.32 0.59 0.44 Latency B C 0.45 0.42 0.39 0.77 0.39 0.32 0.43 0.47 Latency B C 0.33 0.31 0.02 0.54 0.29 0.04 0.32 0.33 Latency B C 0.26 0.30 0.0 0.41 0.28 0.0 0.30 0.28
D 0.81 0.49 0.51 0.44 D 0.37 0.37 0.40 0.44 D 0.25 0.20 0.30 0.04 D 0.26 0.22 0.26 0.0
Appendix C
FitnessDirect model training
Application Direct (raw) Total used
Samples 200
Iters 3
First sample 0 100
Last sample 99
Table C.1: Multiple training samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Array A 0.48 102.18 119.88 2421.29 2983.15 8.69 3.13 30.42 280.10 25.03 25.14 17.38
Array B 0.48 102.12 119.54 2467.32 2973.14 3.02 4.92 30.05 360.82 25.27 11.40 40.74
Array C 0.48 102.40 120.06 2445.39 3025.42 4.37 3.51 12.93 116.16 81.70 77.46 60.74
Array D 0.48 102.06 119.65 2432.69 2979.56 3.47 4.37 37.42 442.60 22.75 14.54 28.27
MAD 0.0 0.1 0.2 14.7 17.6 1.9 0.7 7.4 101.8 21.5 22.7 14.0
COV 0.2% 0.1% 0.2% 0.7% 0.7% 46.0% 17.6% 32.6% 40.2% 64.2% 83.0% 43.8%
Max. Diff. 0.0% 0.3% 0.4% 1.9% 1.8% 187.8% 57.2% 189.4% 281.0% 259.1% 579.5% 249.5%
Table C.2: Workload characteristics and performance are measured for each sample, on each storage device. The average value for each measurement is reported in this table. The mean absolute deviation (MAD), coefficient of variation (COV), and maximum relative differences of these averages are also reported; these metrics quantify how the averages change among the storage devices. For example, the mean average deviation of the averages for Read latency (ms) is 13.96, their coeefficient of variation is 43.806%, and the maximum relative difference is 249.48%.
45
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 0.1% 0.1% 0.0% 0.0% 0.0% 0.2% 0.5% 1.0% 0.8% 0.8% 0.6% 0.4%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 0.2% 0.1% 0.0% 0.7% 0.6% 0.4% 0.0% 1.2% 1.2% 0.5% 0.1% 0.6%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 0.2% 0.1% 0.1% 1.3% 1.1% 0.4% 0.5% 1.2% 1.4% 1.1% 1.2% 1.0%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 0.0% 0.1% 0.1% 0.9% 0.3% 0.3% 0.4% 2.1% 1.7% 1.0% 0.0% 1.6%
Array A Min 25% 0.0 0.2 0.0 3.0 0.0 61.0 0.0 0.0 0.0 1476.0 0.0 6.0 0.0 1.1 3.5 15.7 102.0 148.0 1.2 9.5 0.0 6.8 0.0 8.2 Array B Min 25% 0.0 0.2 0.0 3.0 0.0 61.0 0.0 0.0 0.0 1465.0 0.0 1.0 0.0 1.4 5.4 17.8 68.0 175.0 0.8 9.3 0.0 0.8 0.0 14.6 Array C Min 25% 0.0 0.2 0.0 3.0 0.0 61.0 0.0 0.0 0.0 1475.0 0.0 1.0 0.0 1.1 0.3 6.1 34.0 59.0 1.0 25.5 0.0 18.6 0.0 17.8 Array D Min 25% 0.0 0.2 0.0 3.0 0.0 61.0 0.0 0.0 0.0 1480.0 0.0 1.0 0.0 1.5 7.0 21.2 92.0 191.0 0.7 8.4 0.0 0.5 0.0 11.1
50% 0.6 101.0 115.0 2567.0 3381.0 6.8 1.9 26.9 220.0 16.0 13.6 13.6
75% 0.8 171.1 192.2 4116.0 4428.0 10.3 3.8 39.2 320.0 28.5 25.8 21.5
90% 1.0 217.1 235.1 4762.0 4986.0 17.6 6.5 54.8 451.0 62.5 72.3 36.6
Max 1.0 251.3 255.3 5525.0 5662.0 37.0 15.9 103.4 1184.0 138.6 165.7 67.0
Mean 0.5 102.2 119.9 2421.3 2983.2 8.7 3.1 30.4 280.1 25.0 25.1 17.4
50% 0.6 101.6 115.0 2550.0 3378.0 1.2 2.9 26.4 243.0 15.0 4.0 21.6
75% 0.8 171.0 192.3 4182.0 4396.0 2.5 6.4 39.4 332.0 27.9 7.9 41.8
90% 1.0 217.0 235.2 4917.0 4965.0 7.0 12.5 49.3 423.0 53.9 24.6 105.3
Max 1.0 251.0 255.1 5589.0 5722.0 31.9 18.4 91.3 9477.0 134.5 135.0 264.1
Mean 0.5 102.1 119.5 2467.3 2973.1 3.0 4.9 30.1 360.8 25.3 11.4 40.7
50% 0.6 103.1 115.1 2644.0 3449.0 1.6 2.0 9.5 77.0 43.7 40.4 31.5
75% 0.8 171.3 192.6 4143.0 4370.0 5.4 5.2 12.7 113.0 90.6 78.8 66.9
90% 1.0 216.7 235.0 4848.0 5097.0 11.6 9.0 17.5 166.0 190.3 184.6 156.4
Max 1.0 251.0 255.5 5775.0 5885.0 32.0 15.9 105.3 1229.0 578.0 583.5 423.9
Mean 0.5 102.4 120.1 2445.4 3025.4 4.4 3.5 12.9 116.2 81.7 77.5 60.7
50% 0.6 101.9 115.3 2528.0 3401.0 1.2 3.0 31.2 263.0 12.2 2.5 19.3
75% 0.8 171.1 192.1 4158.0 4398.0 3.2 5.9 47.2 387.0 21.1 7.3 43.1
90% 1.0 217.0 235.2 4813.0 4961.0 8.5 10.1 73.4 633.0 47.1 35.4 61.6
Max 1.0 251.0 255.3 5519.0 5692.0 32.0 29.0 111.0 9133.0 168.4 180.4 134.8
Mean 0.5 102.1 119.7 2432.7 2979.6 3.5 4.4 37.4 442.6 22.8 14.5 28.3
Table C.3: Workload characteristics and performance are measured for each sample, on each storage device. The minimum value, percentiles, maximum value and average are reported for each measurement. In addition, the relative difference between the average performance of the best and second-to-best iteration is reported. This value quantifies the change in a given measurement across multiple runs of the same sample on the same storage device.
47 120
0.8
100 Bandwidth (MB/sec)
1
0.6 0.4
Array A Array B Array C Array D
80 60 40
90
20 0
100
60
50
40
30
20
0
10
0
80
Array A Array B Array C Array D
0.2
70
Fraction
APPENDIX C. FITNESSDIRECT MODEL TRAINING
0
10
20
30
Bandwidth (MB/sec) 10000 9000
0.8
8000
Throughput (IO/sec)
1 0.9
0.6 0.5 0.4 0.3
500
400
300
200
100
0
0
4000 3000 2000 1000 0 0
1
600
Latency (ms)
0.6 0.5 0.4 Array A Array B Array C Array D
Latency (ms)
400 300 200
350
100 300
200
150
100
50
0
0
250
Fraction
0.7
10 20 30 40 50 60 70 80 90 100 Sample number
Array A Array B Array C Array D
500
0.8
0.1
Array A Array B Array C Array D
5000
0.9
0.2
90 100
6000
Throughput (IO/sec)
0.3
80
7000
900
0.1
800
0.2
700
Array A Array B Array C Array D 600
Fraction
0.7
40 50 60 70 Sample number
0 0
10
20
30
40 50 60 70 Sample number
80
90 100
Figure C.1: The cumulative distribution of performance is shown (top). In addition, the performance of each array is shown (bottom), sorted by the performance of Array C.
48
Throughput (IO/sec) on Array A
<1
1
[11
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
[18 [25 [32 [39 [46 >= 5 , , , , , 8) 25) 32) 39) 46) 53) 3
0.3 0.25
0.1
<1
[ [ [ [ [ [ > 77 177, 252, 327, 402, 477, 552, =62 7 25 3 4 4 5 6 2) 27) 02) 77) 52) 27)
Bandwidth (MB/sec) on Array B
Fraction
Fraction
0.16
0.08 0.06 0.04
<1
2
[12
0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0
0.35 0.3
24
[19 [25 [32 [38 [45 >= 5 , , , , , 9) 25) 32) 38) 45) 51) 1
0.1 0
[9,
12
)
[12
<6
1
[15 [18 >= 2 , , 5) 18) 21) 1
,1
[61
[87 [11 [14 [16 [19 >= 2 , 4 0 7 4 7) 114 , 14 , 16 , 19 , 22 20 ) 0) 7) 4) 0)
Fraction
Fraction
0.2 0.15 0.1 0.05 0
8
[18
[30 [41 [52 [64 [75 >= 8 , , , , , 0) 41) 52) 64) 75) 87) 7
,3
[41
[80 [12 [15 [19 [23 >= 2 , 0 9 9 9 0) 120 , 15 , 19 , 23 , 27 78 ) 9) 9) 9) 8)
,8
Latency (ms) on Array D
0.3
<1
<4
1
0.25
0.1
[24 [36 [47 [59 [71 >= 8 , , , , , 4) 36) 47) 59) 71) 82) 2
,2
Throughput (IO/sec) on Array D
0.2
0.05
0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
,8
0.35
0.15
[12
Latency (ms) on Array C
0.05
9)
<1
2
Fraction
Fraction
Fraction
0.2 0.15
Bandwidth (MB/sec) on Array D
Fraction
0
Throughput (IO/sec) on Array C
0.25
0
[1 [2 [2 [3 [4 > 4, 80, 36, 92, 47, 03, =459 18 2 2 3 4 4 0) 36) 92) 47) 03) 59)
0.3
[6,
0.2 0.15 0.05
[12
0.25
6)
0.25
0.1
<1
,1
[3,
[21 [31 [41 [51 [61 >= 7 , , , , , 1) 31) 41) 51) 61) 71) 1
,2
Latency (ms) on Array B
0.35
<3
[11
0.4
Bandwidth (MB/sec) on Array C 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0
<1
1
Fraction
0.18
0.1
0
Throughput (IO/sec) on Array B
0.2
0.12
0.2 0.15
0.05
,1
0.14
Latency (ms) on Array A 0.35
Fraction
Fraction
Fraction
Bandwidth (MB/sec) on Array A 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<1
[ [ [ [ [ [ > 87 187, 283, 378, 474, 569, 664, =76 0 28 3 4 5 6 7 3) 78) 74) 69) 64) 60)
0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
Figure C.2: Probability distributions of performance.
1
[11
[21 [31 [41 [51 [62 >= 7 , , , , , 1) 31) 41) 51) 62) 72) 2
,2
49
APPENDIX C. FITNESSDIRECT MODEL TRAINING
Array A to Array B (min 0.27, max 8.00) 0.3
Array A to Array C (min 0.08, max 1.87) 0.25
0.25 0.2
Array A to Array D (min 0.71, max 7.71) 0.25
0.2
0.2
0.15
0.15
0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .45 0.45 0.63 0.81 0.99 1.17 1.35 =1.5 ,0 , , , , , .63 0.81 0.99 1.17 1.35 1.53 3 ) ) ) ) ) )
0.1
0.1
0.05
0.05
0
<0
[ [ [ [ [ [ > .15 0.15 0.22 0.29 0.36 0.43 0.50 =0.5 ,0 , , , , , .22 0.29 0.36 0.43 0.50 0.57 7 ) ) ) ) ) )
Array B to Array A (min 0.12, max 3.77) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .29 0.29 0.45 0.62 0.78 0.95 1.11 =1.2 ,0 , , , , , .45 0.62 0.78 0.95 1.11 1.28 8 ) ) ) ) ) )
0.25 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .06 1.06 1.58 2.10 2.62 3.14 3.67 =4.1 ,1 , , , , , .58 2.10 2.62 3.14 3.67 4.19 9 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .12 0.12 0.22 0.32 0.43 0.53 0.64 =0.7 ,0 , , , , , .22 0.32 0.43 0.53 0.64 0.74 4 ) ) ) ) ) )
0.25 0.2
<0
[ [ [ [ [ [ > .90 0.90 1.04 1.18 1.31 1.45 1.59 =1.7 ,1 , , , , , .04 1.18 1.31 1.45 1.59 1.73 3 ) ) ) ) ) )
Array C to Array D (min 0.97, max 92.28) 0.3 0.25 0.2 0.15 0.1 0.05
<0
[ [ [ [ [ [ > .89 0.89 1.55 2.20 2.85 3.50 4.15 =4.8 ,1 , , , , , .55 2.20 2.85 3.50 4.15 4.80 0 ) ) ) ) ) )
0
<1
[ [ [ [ [ [ > .69 1.69 2.41 3.13 3.85 4.57 5.29 =6.0 ,2 , , , , , .41 3.13 3.85 4.57 5.29 6.01 1 ) ) ) ) ) )
Array D to Array B (min 0.22, max 1.31)
Array D to Array C (min 0.01, max 1.03)
0.25
0.3
0.2
0.25 0.2
0.15
0.15
0.15 0.1
0.1
0.1
0.05
0.05 0
[ [ [ [ [ [ > .88 0.88 1.04 1.21 1.37 1.54 1.71 =1.8 ,1 , , , , , .04 1.21 1.37 1.54 1.71 1.87 7 ) ) ) ) ) )
Array B to Array D (min 0.76, max 4.50) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0
Array C to Array B (min 0.24, max 95.72) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02
Array D to Array A (min 0.13, max 1.41) 0.3
<0
Array B to Array C (min 0.01, max 4.13) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array A (min 0.54, max 11.97) 0.3
0
<0
[ [ [ [ [ [ > .27 0.27 0.40 0.54 0.67 0.81 0.94 =1.0 ,0 , , , , , .40 0.54 0.67 0.81 0.94 1.08 8 ) ) ) ) ) )
0
0.05
<0
[ [ [ [ [ [ > .34 0.34 0.46 0.59 0.71 0.83 0.95 =1.0 ,0 , , , , , .46 0.59 0.71 0.83 0.95 1.07 7 ) ) ) ) ) )
0
<0
[ [ [ [ [ [ > .08 0.08 0.15 0.22 0.30 0.37 0.44 =0.5 ,0 , , , , , .15 0.22 0.30 0.37 0.44 0.51 1 ) ) ) ) ) )
Figure C.3: Each graph shows the probability distribution of the Bandwidth relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
50
Array A to Array B (min 0.27, max 8.00) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array A to Array C (min 0.08, max 1.86) 0.25
<0
[ [ [ [ [ [ > .46 0.46 0.65 0.84 1.03 1.22 1.41 =1.6 ,0 , , , , , .65 0.84 1.03 1.22 1.41 1.60 0 ) ) ) ) ) )
0.2
0.2
0.15
0.15
0.1
0.1
0.05
0.05
0
<0
[ [ [ [ [ [ > .15 0.15 0.22 0.29 0.36 0.43 0.50 =0.5 ,0 , , , , , .22 0.29 0.36 0.43 0.50 0.57 7 ) ) ) ) ) )
Array B to Array A (min 0.12, max 3.76) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .29 0.29 0.45 0.62 0.78 0.95 1.11 =1.2 ,0 , , , , , .45 0.62 0.78 0.95 1.11 1.28 8 ) ) ) ) ) )
0.25 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .06 1.06 1.58 2.11 2.63 3.15 3.68 =4.2 ,1 , , , , , .58 2.11 2.63 3.15 3.68 4.20 0 ) ) ) ) ) )
0.2
[ [ [ [ [ [ > .89 0.89 1.06 1.24 1.42 1.60 1.77 =1.9 ,1 , , , , , .06 1.24 1.42 1.60 1.77 1.95 5 ) ) ) ) ) )
Array B to Array D (min 0.76, max 4.50)
0.2 0.15 0.1 0.05
<0
[ [ [ [ [ [ > .12 0.12 0.22 0.32 0.43 0.53 0.64 =0.7 ,0 , , , , , .22 0.32 0.43 0.53 0.64 0.74 4 ) ) ) ) ) )
0
Array C to Array B (min 0.24, max 96.70) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02
<0
[ [ [ [ [ [ > .90 0.90 1.04 1.17 1.31 1.45 1.59 =1.7 ,1 , , , , , .04 1.17 1.31 1.45 1.59 1.72 2 ) ) ) ) ) )
Array C to Array D (min 0.97, max 93.19) 0.3 0.25 0.2 0.15 0.1 0.05
<0
[ [ [ [ [ [ > .90 0.90 1.55 2.20 2.86 3.51 4.16 =4.8 ,1 , , , , , .55 2.20 2.86 3.51 4.16 4.81 1 ) ) ) ) ) )
0
<1
[ [ [ [ [ [ > .70 1.70 2.42 3.14 3.87 4.59 5.31 =6.0 ,2 , , , , , .42 3.14 3.87 4.59 5.31 6.04 4 ) ) ) ) ) )
Array D to Array B (min 0.22, max 1.31) 0.25
0.25
<0
0.25
Array D to Array A (min 0.13, max 1.41) 0.3
0
Array B to Array C (min 0.01, max 4.12) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array A (min 0.54, max 12.08) 0.3
Array A to Array D (min 0.71, max 7.71) 0.25
Array D to Array C (min 0.01, max 1.03) 0.25
0.2
0.2
0.15
0.15
0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .27 0.27 0.40 0.54 0.68 0.81 0.95 =1.0 ,0 , , , , , .40 0.54 0.68 0.81 0.95 1.09 9 ) ) ) ) ) )
0.1
0.1
0.05
0.05
0
<0
[ [ [ [ [ [ > .34 0.34 0.46 0.59 0.71 0.83 0.95 =1.0 ,0 , , , , , .46 0.59 0.71 0.83 0.95 1.07 7 ) ) ) ) ) )
0
<0
[ [ [ [ [ [ > .08 0.08 0.15 0.23 0.30 0.37 0.44 =0.5 ,0 , , , , , .15 0.23 0.30 0.37 0.44 0.51 1 ) ) ) ) ) )
Figure C.4: Each graph shows the probability distribution of the Throughput relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
51
APPENDIX C. FITNESSDIRECT MODEL TRAINING
Array A to Array B (min 0.12, max 4.02) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .28 0.28 0.45 0.61 0.78 0.95 1.11 =1.2 ,0 , , , , , .45 0.61 0.78 0.95 1.11 1.28 8 ) ) ) ) ) )
Array A to Array C (min 0.50, max 12.20) 0.3
0.25
0.25
0.2
0.2
0.15
0.15
0.1
0.1
0.05
0.05
0
Array B to Array A (min 0.25, max 8.59) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .43 0.43 0.62 0.80 0.99 1.17 1.36 =1.5 ,0 , , , , , .62 0.80 0.99 1.17 1.36 1.54 4 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .99 0.99 1.49 1.98 2.48 2.98 3.47 =3.9 ,1 , , , , , .49 1.98 2.48 2.98 3.47 3.97 7 ) ) ) ) ) )
0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .16 0.16 0.23 0.31 0.38 0.46 0.53 =0.6 ,0 , , , , , .23 0.31 0.38 0.46 0.53 0.61 1 ) ) ) ) ) )
0.1 0.05
<0
[ [ [ [ [ [ > .89 0.89 1.56 2.24 2.91 3.58 4.26 =4.9 ,1 , , , , , .56 2.24 2.91 3.58 4.26 4.93 3 ) ) ) ) ) )
0
Array C to Array D (min 0.01, max 0.99)
0.2
0.1 0.05
<0
[ [ [ [ [ [ > .12 0.12 0.23 0.35 0.46 0.57 0.68 =0.7 ,0 , , , , , .23 0.35 0.46 0.57 0.68 0.79 9 ) ) ) ) ) )
0
Array D to Array B (min 0.77, max 5.15)
0.1
[ [ [ [ [ [ > .89 0.89 1.06 1.22 1.39 1.55 1.72 =1.8 ,1 , , , , , .06 1.22 1.39 1.55 1.72 1.88 8 ) ) ) ) ) )
[ [ [ [ [ [ > .32 0.32 0.45 0.57 0.70 0.82 0.95 =1.0 ,0 , , , , , .45 0.57 0.70 0.82 0.95 1.08 8 ) ) ) ) ) )
0.15
0.05
<0
<0
0.25
0.15 0.1
0
Array C to Array B (min 0.01, max 4.61) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.15
0.05
Array B to Array D (min 0.19, max 1.29)
0.2
0.2
0.2
[ [ [ [ [ [ > .25 0.25 0.39 0.54 0.68 0.82 0.96 =1.1 ,0 , , , , , .39 0.54 0.68 0.82 0.96 1.10 0 ) ) ) ) ) )
0.15
0.25
0.25
<0
0.25
Array D to Array A (min 0.73, max 9.18) 0.3
0
Array B to Array C (min 0.22, max 104.81) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02
Array C to Array A (min 0.08, max 2.02) 0.3
Array A to Array D (min 0.11, max 1.37)
0.3
0
<0
[ [ [ [ [ [ > .93 0.93 1.08 1.23 1.38 1.54 1.69 =1.8 ,1 , , , , , .08 1.23 1.38 1.54 1.69 1.84 4 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .08 0.08 0.15 0.22 0.29 0.36 0.43 =0.5 ,0 , , , , , .15 0.22 0.29 0.36 0.43 0.50 0 ) ) ) ) ) )
Array D to Array C (min 1.01, max 111.99) 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<1
[ [ [ [ [ [ > .74 1.74 2.48 3.22 3.96 4.70 5.44 =6.1 ,2 , , , , , .48 3.22 3.96 4.70 5.44 6.18 8 ) ) ) ) ) )
Figure C.5: Each graph shows the probability distribution of the Latency relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
52
0.9
1
0.8
0.9 0.8
0.4
Array A Array B Array C Array D
0.3 150
Write size (KB) 1
0.9
0.9
0.8
0.8
0.7
0.7
0.5 Array A Array B Array C Array D
0.4
Write queue
3000
0
0.2 2000
15
10
5
0.2
0.3
25
0.3
20
Array A Array B Array C Array D
1000
0.4
Write jump (MB)
Figure C.6: The cumulative distribution of workload characteristics.
6000
0.5
0.6
5000
0.6
4000
Fraction
1
0
Fraction
Write fraction
100
0
1
0.2 0.9
0.6
0.5
0.4
0.3
0.2
0.1
0
0.2
0.8
Array A Array B Array C Array D
0.3
0.5
50
0.4
0.6
250
0.5
0.7
200
Fraction
0.6
0.7
Fraction
0.7
53
APPENDIX C. FITNESSDIRECT MODEL TRAINING
Overall Predictor Read queue Write queue Read jump Read size Write size Write jump Write fraction
Score 1.00 0.76 0.44 0.41 0.38 0.27 0.14
Overall Predictor Read queue Write queue Read size Write size Read jump Write jump Write fraction
Score 1.00 0.86 0.47 0.38 0.37 0.21 0.08
Overall Predictor Latency Bandwidth Throughput Read queue Read latency Write fraction Write latency Write queue Read jump Read size Write jump Write size
Score 1.00 0.83 0.59 0.19 0.17 0.17 0.16 0.16 0.14 0.13 0.11 0.07
Overall Predictor Write fraction Latency Bandwidth Throughput Write jump Write queue Write latency Read latency Read jump Read queue Write size Read size
Score 1.00 0.89 0.64 0.49 0.29 0.27 0.26 0.23 0.23 0.15 0.12 0.12
Absolute Bandwidth Throughput Predictor Score Predictor Score Read queue 1.00 Read queue 1.00 Read size 0.63 Write size 0.80 Read jump 0.56 Write jump 0.71 Write size 0.54 Read jump 0.61 Write queue 0.44 Write fraction 0.49 Write jump 0.27 Write queue 0.43 Write fraction 0.05 Read size 0.25 Relative Bandwidth Throughput Predictor Score Predictor Score Read queue 1.00 Read queue 1.00 Write queue 0.71 Read size 0.68 Read size 0.62 Write size 0.51 Read jump 0.57 Read jump 0.42 Write size 0.52 Write jump 0.40 Write jump 0.24 Write queue 0.36 Write fraction 0.12 Write fraction 0.10 Relative Performance Bandwidth Throughput Predictor Score Predictor Score Bandwidth 1.00 Throughput 1.00 Latency 0.24 Bandwidth 0.26 Write fraction 0.15 Latency 0.11 Write jump 0.12 Read queue 0.11 Throughput 0.11 Write queue 0.08 Read size 0.10 Write size 0.05 Read queue 0.10 Write fraction 0.05 Read jump 0.09 Read size 0.05 Write queue 0.09 Write jump 0.04 Read latency 0.08 Read jump 0.04 Write size 0.05 Write latency 0.02 Write latency 0.04 Read latency 0.01 Relative Fitness Bandwidth Throughput Predictor Score Predictor Score Write fraction 1.00 Write fraction 1.00 Latency 0.93 Latency 0.89 Bandwidth 0.80 Bandwidth 0.84 Throughput 0.35 Write queue 0.38 Read jump 0.32 Throughput 0.32 Write queue 0.29 Write latency 0.28 Write latency 0.26 Read jump 0.25 Write jump 0.23 Read latency 0.23 Read latency 0.18 Write jump 0.23 Write size 0.16 Read queue 0.15 Read size 0.11 Write size 0.14 Read queue 0.11 Read size 0.05
Latency Predictor Write queue Read queue Read size Read jump Write jump Write fraction Write size
Score 1.00 0.86 0.24 0.23 0.10 0.08 0.07
Latency Predictor Write queue Read queue Read size Write size Read jump Write jump Write fraction
Score 1.00 0.80 0.19 0.16 0.16 0.06 0.04
Latency Predictor Latency Write latency Read latency Bandwidth Read queue Write queue Read jump Write fraction Read size Write size Write jump Throughput
Score 1.00 0.16 0.16 0.10 0.09 0.08 0.08 0.06 0.05 0.02 0.02 0.01
Latency Predictor Write fraction Latency Throughput Write jump Bandwidth Read latency Write latency Read size Read queue Write queue Read jump Write size
Score 1.00 0.85 0.81 0.40 0.28 0.28 0.24 0.20 0.20 0.15 0.13 0.07
Table C.4: The normalized importance measure of each predictor.
Model Absolute Relative Relative Performance Relative Fitness
Overall 11.9 11.6 13.1 10.6
Absolute Array Array Array Array Relative Array Array Array Array Relative Performance Array Array Array Array Relative Fitness Array Array Array Array
Absolute Array A Array B Array C Array D
A 20.0 -
Relative Array A Array B Array C Array D
A 20.0 20.0 19.0 21.0
Relative Performance Array A Array B Array C Array D
A 18.0 18.0 4.0 21.0
Relative Fitness Array A Array B Array C Array D
A 1 12.0 14.0 16.0
Bandwidth B C 15.0 9.0 Bandwidth B C 10.0 13.0 15.0 1.0 15.0 9.0 13.0 1.0 Bandwidth B C 18.0 17.0 15.0 5.0 16.0 19.0 23.0 22.0 Bandwidth B C 1.0 3.0 1 10.0 17.0 1 18.0 18.0
Bandwidth 14.5 11.5 15.8 11.2
Pairwise Array 14.0 Array 14.0 14.7 15.0 13.0 Array 16.7 13.3 9.0 18.3 Array 1 13.0 17.0 13.3
Array 9.7 Array 10.3 9.7 17.3 14.7 Array 12.3 16.7 18.3 17.3 Array 6.3 1 16.0 13.3
D 14.0
A 4.0 -
D 14.0 4.0 7.0 14.0
A 4.0 6.0 11.0 6.0
D 16.0 8.0 22.0 16.0
A 16.0 5.0 2.0 17.0
D 5.0 4.0 17.0 1
A 1 16.0 19.0 17.0
Throughput 7.8 9.1 6.8 10.4
Array 15.0 Array 14.3 10.7 15.0 9.0 Array 13.3 9.0 16.3 9.3 Array 2.7 11.0 1 11.0
Throughput B C 5.0 20.0 Throughput B C 3.0 15.0 5.0 15.0 20.0 20.0 18.0 12.0 Throughput B C 1.0 3.0 17.0 1.0 18.0 16.0 11.0 1.0 Throughput B C 1.0 3.0 1 11.0 12.0 1 10.0 5.0
Latency 13.5 14.2 16.6 10.2
Array 9.0 Array 7.3 3.7 9.0 9.0 Array 8.7 13.0 14.7 17.0 Array 4.7 5.7 13.3 1
D 2.0
A 18.0 -
D 1.0 1.0 1.0 2.0
A 18.0 18.0 15.0 12.0
D 1.0 18.0 3.0 18.0
A 16.0 17.0 21.0 17.0
D 6.0 10.0 15.0 1
A 1 11.0 18.0 7.0
Table C.5: Tree sizes (leaf nodes) and their averages.
Latency B C 9.0 16.0 Latency B C 18.0 15.0 9.0 16.0 17.0 16.0 13.0 14.0 Latency B C 18.0 20.0 18.0 21.0 21.0 14.0 18.0 5.0 Latency B C 17.0 2.0 1 12.0 19.0 1 12.0 10.0
D 11.0 D 7.0 6.0 19.0 11.0 D 9.0 13.0 19.0 17.0 D 3.0 3.0 8.0 1
55
APPENDIX C. FITNESSDIRECT MODEL TRAINING
Bandwidth error (FitnessDirect) Median relative error (Avg.)
Median relative error (Avg.)
Overall error (FitnessDirect) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
0.4 0.35 0.3 0.25 0.2 0.15 0.1
70
30 AM (i=j) AM
RF
0.3 0.28 0.26 0.24 0.22 0.2 0.18 0.16 0.14 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
RM RM’
70 RF
Latency error (FitnessDirect) Median relative error (Avg.)
Median relative error (Avg.)
Throughput error (FitnessDirect)
40 50 60 Samples used for training (%)
70 RF
0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
Figure C.7: Error vs. training. The training set size varies from 25% of the collected samples to 75%; the remaining samples are used for testing. For each training set size, the Median relative error is calculated for each pairing of devices. They are then averaged to produce the values shown.
56
Bandwidth error (FitnessDirect) Mean relative error (Avg.)
Mean relative error (Avg.)
Overall error (FitnessDirect) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2
70
30
RF
AM (i=j) AM
30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
RM RM’
70 RF
Latency error (FitnessDirect)
0.6 0.55 0.5 0.45 0.4 0.35 0.3 0.25 0.2
Mean relative error (Avg.)
Mean relative error (Avg.)
Throughput error (FitnessDirect)
40 50 60 Samples used for training (%)
70 RF
1.8 1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
Figure C.8: Error vs. training. The training set size varies from 25% of the collected samples to 75%; the remaining samples are used for testing. For each training set size, the Mean relative error is calculated for each pairing of devices. They are then averaged to produce the values shown.
Appendix D
FitnessBuffered model training
Application Buffered (sd) Total used
Samples 200
Iters 3
First sample 0 100
Last sample 99
Table D.1: Multiple training samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Array A 0.31 99.62 63.82 255.22 2057.00 26.36 5.07 27.54 372.61 34.78 66.61 13.78
Array B 0.31 100.34 63.94 237.39 2061.67 26.24 5.07 20.96 280.07 35.61 52.38 21.20
Array C 0.30 104.30 63.77 302.71 2116.93 24.97 4.79 16.18 225.89 74.23 146.08 25.35
Array D 0.31 99.77 63.77 252.20 2055.77 25.96 5.06 28.83 421.02 28.05 36.67 16.38
MAD 0.0 1.6 0.1 20.4 22.0 0.5 0.1 4.8 71.9 15.5 35.3 4.1
COV 1.8% 1.9% 0.1% 9.4% 1.2% 2.1% 2.4% 21.9% 23.5% 42.1% 55.9% 23.2%
Max. Diff. 3.3% 4.7% 0.3% 27.5% 3.0% 5.6% 5.8% 78.2% 86.4% 164.6% 298.4% 84.0%
Table D.2: Workload characteristics and performance are measured for each sample, on each storage device. The average value for each measurement is reported in this table. The mean absolute deviation (MAD), coefficient of variation (COV), and maximum relative differences of these averages are also reported; these metrics quantify how the averages change among the storage devices. For example, the mean average deviation of the averages for Read latency (ms) is 4.10, their coeefficient of variation is 23.199%, and the maximum relative difference is 83.96%.
57
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 1.6% 0.4% 0.2% 1.8% 0.3% 0.8% 0.4% 1.5% 0.1% 0.4% 0.2% 0.0%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 0.3% 0.1% 0.3% 12.7% 0.0% 3.1% 1.7% 1.9% 1.6% 1.6% 2.4% 0.5%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 0.5% 0.0% 0.1% 135.5% 0.3% 0.6% 3.8% 1.8% 4.1% 1.3% 0.3% 2.0%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 3.2% 0.1% 0.2% 1.0% 0.3% 1.0% 0.5% 3.5% 3.6% 1.7% 0.7% 3.6%
Array A 25% 0.1 7.8 30.0 1.0 1353.0 12.9 1.5 16.7 224.0 16.0 43.4 6.4 Array B Min 25% 0.0 0.1 0.0 8.0 0.0 29.9 0.0 1.0 0.0 1409.0 0.0 13.6 0.0 1.5 4.7 13.2 69.0 187.0 3.1 14.4 0.0 25.1 0.0 11.0 Array C Min 25% 0.0 0.1 0.0 7.6 0.0 30.2 0.0 1.0 0.0 1442.0 0.0 11.7 0.0 1.5 3.0 8.8 59.0 117.0 1.6 27.2 0.0 93.9 0.0 9.8 Array D Min 25% 0.0 0.1 0.0 7.3 0.0 29.3 0.0 1.0 0.0 1353.0 0.0 14.3 0.0 1.5 5.2 14.6 99.0 203.0 1.2 11.5 0.0 5.2 0.0 9.3
Min 0.0 0.0 0.0 0.0 0.0 0.0 0.0 6.7 121.0 2.3 0.0 0.0
50% 0.3 118.6 59.8 220.0 2267.0 23.9 2.6 23.2 296.0 28.6 68.6 9.9
75% 0.4 168.4 94.1 340.0 2926.0 41.2 6.6 34.6 409.0 45.6 92.7 17.9
90% 0.6 188.8 119.9 650.0 3438.0 54.8 10.8 46.0 521.0 62.4 121.4 28.9
Max 1.0 227.4 255.1 1166.0 4359.0 64.0 26.9 87.2 4219.0 135.2 162.4 54.9
Mean 0.3 99.6 63.8 255.2 2057.0 26.4 5.1 27.5 372.6 34.8 66.6 13.8
50% 0.3 119.5 60.1 247.0 2259.0 25.7 2.8 18.4 233.0 21.8 41.8 15.3
75% 0.5 167.6 94.7 332.0 2870.0 38.6 7.3 27.1 309.0 35.5 61.8 21.4
90% 0.6 191.9 120.3 520.0 3406.0 55.8 10.9 32.6 391.0 64.6 118.1 42.7
Max 1.0 224.8 252.3 1381.0 4386.0 64.0 24.3 56.9 2789.0 264.0 271.3 95.6
Mean 0.3 100.3 63.9 237.4 2061.7 26.2 5.1 21.0 280.1 35.6 52.4 21.2
50% 0.3 126.8 59.9 56.0 2381.0 22.3 2.7 12.7 148.0 50.4 136.1 16.3
75% 0.4 174.3 94.3 241.0 3029.0 41.0 6.6 16.7 188.0 76.8 183.0 32.2
90% 0.6 195.7 121.2 330.0 3447.0 50.9 10.5 23.9 290.0 154.9 332.5 62.5
Max 1.0 238.2 255.3 16383.0 4360.0 64.0 22.9 96.8 3610.0 485.7 489.2 123.5
Mean 0.3 104.3 63.8 302.7 2116.9 25.0 4.8 16.2 225.9 74.2 146.1 25.4
50% 0.3 118.7 60.0 227.0 2276.0 23.8 2.6 21.9 288.0 16.5 28.9 12.2
75% 0.4 167.0 95.1 350.0 2916.0 36.3 7.7 38.2 408.0 27.4 42.5 18.6
90% 0.6 188.7 121.1 561.0 3381.0 54.5 10.9 51.0 625.0 44.8 71.1 33.3
Max 1.0 226.7 255.8 1358.0 4456.0 64.0 24.8 103.6 7850.0 211.9 214.8 55.7
Mean 0.3 99.8 63.8 252.2 2055.8 26.0 5.1 28.8 421.0 28.0 36.7 16.4
Table D.3: Workload characteristics and performance are measured for each sample, on each storage device. The minimum value, percentiles, maximum value and average are reported for each measurement. In addition, the relative difference between the average performance of the best and second-to-best iteration is reported. This value quantifies the change in a given measurement across multiple runs of the same sample on the same storage device.
59
APPENDIX D. FITNESSBUFFERED MODEL TRAINING 1
120
Bandwidth (MB/sec)
0.8 0.6 0.4 Array A Array B Array C Array D 50
40
30
20
10
0
0
80 60 40 20 0 0
70
0.2
60
Fraction
Array A Array B Array C Array D
100
10
20
30
Bandwidth (MB/sec) 1
8000
0.9
7000 Throughput (IO/sec)
0.8 0.6 0.5 0.4 0.3
Array A Array B Array C Array D 500
400
300
200
100
0
0
4000 3000 2000 1000 0 0
1
500
Latency (ms)
0.4 Array A Array B Array C Array D
350 300 250 200 150 100 50
250
150
100
50
0
Latency (ms)
200
Fraction
400
0.6
10 20 30 40 50 60 70 80 90 100 Sample number Array A Array B Array C Array D
450 0.8
0
90 100
5000
Throughput (IO/sec)
0.2
80
Array A Array B Array C Array D
6000
800
0.1
700
0.2
600
Fraction
0.7
40 50 60 70 Sample number
0 0
10
20
30
40 50 60 70 Sample number
80
90 100
Figure D.1: The cumulative distribution of performance is shown (top). In addition, the performance of each array is shown (bottom), sorted by the performance of Array C.
60
Throughput (IO/sec) on Array A
0.2
Fraction
Fraction
0.25
0.15 0.1 0.05 0
<1
3
[13
0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
[19 [25 [32 [38 [44 >= 5 , , , , , 9) 25) 32) 38) 44) 50) 0
<1
[ [ [ [ [ [ > 85 185, 248, 312, 376, 440, 504, =56 7 24 3 3 4 5 5 8) 12) 76) 40) 04) 67)
,1
<9
[9,
13
)
[13
0.3
0.05
11
)
[11
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
[15 [19 [23 [27 >= 3 , , , , 5) 19) 23) 27) 31) 1
0
4
[ [ [ [ [ [ > 04 104, 148, 193, 238, 283, 328, =37 2 14 1 2 2 3 3 8) 93) 38) 83) 28) 72)
0.1
<2
7
[27
[52 [77 [10 [12 [15 >= 1 , , 2 8 3 2) 77) 102 , 12 , 15 , 17 78 ) 8) 3) 8)
,5
Latency (ms) on Array D
Fraction
0.15 0.1
0
[20 [28 [36 [43 [51 >= 5 , , , , , 0) 28) 36) 43) 51) 58) 8
,2
0.2 0.15
0
<1
0.1
[13
[25 [36 [47 [58 [69 >= 7 , , , , , 5) 36) 47) 58) 69) 79) 9
,2
0.05
0.05
3
[14
Latency (ms) on Array C
0.2
<1
<1
Throughput (IO/sec) on Array D
Fraction
Fraction
[1 [2 [2 [3 [3 > 0, 71, 22, 72, 23, 74, =425 17 2 2 3 3 4 1) 22) 72) 23) 74) 25)
0.25
0.25
0.05
0.05
[12
0.25
Bandwidth (MB/sec) on Array D 0.3
0.15
0.2 0.15
0.3
,1
0.2
0.25
0.1
<1
Fraction
Fraction
Fraction
0.1
[7,
Latency (ms) on Array B
Throughput (IO/sec) on Array C
0.25
<7
[22 [33 [43 [53 [63 >= 7 , , , , , 2) 33) 43) 53) 63) 73) 3
,2
0.35
Bandwidth (MB/sec) on Array C 0.3
0
[12
0.4
20
[18 [22 [26 [31 >= 3 , , , , 8) 22) 26) 31) 35) 5
0.15
<1
2
Fraction
0.26 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06
,1
0.2
0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02
Throughput (IO/sec) on Array B
Fraction
Fraction
Bandwidth (MB/sec) on Array B 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
Latency (ms) on Array A
Fraction
Bandwidth (MB/sec) on Array A 0.3
<1
[ [ [ [ [ [ > 87 187, 274, 362, 450, 537, 625, =71 2 27 3 4 5 6 7 4) 62) 50) 37) 25) 12)
0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
Figure D.2: Probability distributions of performance.
5
[15
[29 [42 [56 [70 [84 >= 9 , , , , , 9) 42) 56) 70) 84) 98) 8
,2
61
APPENDIX D. FITNESSBUFFERED MODEL TRAINING
Array A to Array B (min 0.34, max 1.17)
Array A to Array C (min 0.26, max 1.63)
0.25
0.3
0.2
0.25 0.2
0.15
0.15 0.1
0.1
0.05 0
0.05
<0
[ [ [ [ [ [ > .43 0.43 0.52 0.61 0.70 0.79 0.88 =0.9 ,0 , , , , , .52 0.61 0.70 0.79 0.88 0.97 7 ) ) ) ) ) )
0
<0
[ [ [ [ [ [ > .35 0.35 0.45 0.54 0.64 0.74 0.83 =0.9 ,0 , , , , , .45 0.54 0.64 0.74 0.83 0.93 3 ) ) ) ) ) )
Array B to Array A (min 0.86, max 2.92) 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .00 1.00 1.14 1.28 1.42 1.56 1.70 =1.8 ,1 , , , , , .14 1.28 1.42 1.56 1.70 1.84 4 ) ) ) ) ) )
0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .95 0.95 1.29 1.62 1.96 2.30 2.63 =2.9 ,1 , , , , , .29 1.62 1.96 2.30 2.63 2.97 7 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .40 0.40 0.55 0.70 0.84 0.99 1.14 =1.2 ,0 , , , , , .55 0.70 0.84 0.99 1.14 1.29 9 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .68 0.68 0.77 0.87 0.97 1.06 1.16 =1.2 ,0 , , , , , .77 0.87 0.97 1.06 1.16 1.25 5 ) ) ) ) ) )
[ [ [ [ [ [ > .63 0.63 0.75 0.88 1.01 1.14 1.27 =1.4 ,0 , , , , , .75 0.88 1.01 1.14 1.27 1.40 0 ) ) ) ) ) )
Array B to Array D (min 0.88, max 4.41) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .04 1.04 1.20 1.36 1.52 1.68 1.84 =2.0 ,1 , , , , , .20 1.36 1.52 1.68 1.84 2.00 0 ) ) ) ) ) )
Array C to Array B (min 0.27, max 4.01) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .62 0.62 0.97 1.32 1.67 2.02 2.38 =2.7 ,0 , , , , , .97 1.32 1.67 2.02 2.38 2.73 3 ) ) ) ) ) )
Array D to Array A (min 0.58, max 2.01) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06
<0
Array B to Array C (min 0.25, max 3.69) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array A (min 0.61, max 3.86) 0.3
Array A to Array D (min 0.50, max 1.72) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array D (min 0.93, max 5.28) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .28 1.28 1.63 1.99 2.34 2.69 3.04 =3.3 ,1 , , , , , .63 1.99 2.34 2.69 3.04 3.39 9 ) ) ) ) ) )
Array D to Array B (min 0.23, max 1.14) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .32 0.32 0.41 0.51 0.60 0.69 0.79 =0.8 ,0 , , , , , .41 0.51 0.60 0.69 0.79 0.88 8 ) ) ) ) ) )
Array D to Array C (min 0.19, max 1.08) 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<0
[ [ [ [ [ [ > .27 0.27 0.35 0.43 0.51 0.59 0.67 =0.7 ,0 , , , , , .35 0.43 0.51 0.59 0.67 0.76 6 ) ) ) ) ) )
Figure D.3: Each graph shows the probability distribution of the Bandwidth relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
62
Array A to Array B (min 0.37, max 1.17) 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .46 0.46 0.54 0.63 0.71 0.80 0.88 =0.9 ,0 , , , , , .54 0.63 0.71 0.80 0.88 0.97 7 ) ) ) ) ) )
Array A to Array C (min 0.27, max 1.65) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .36 0.36 0.46 0.55 0.64 0.74 0.83 =0.9 ,0 , , , , , .46 0.55 0.64 0.74 0.83 0.93 3 ) ) ) ) ) )
Array B to Array A (min 0.86, max 2.70) 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .99 0.99 1.13 1.27 1.41 1.55 1.69 =1.8 ,1 , , , , , .13 1.27 1.41 1.55 1.69 1.83 3 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .97 0.97 1.33 1.70 2.06 2.43 2.79 =3.1 ,1 , , , , , .33 1.70 2.06 2.43 2.79 3.16 6 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .39 0.39 0.53 0.66 0.80 0.94 1.08 =1.2 ,0 , , , , , .53 0.66 0.80 0.94 1.08 1.21 1 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .64 0.64 0.75 0.85 0.95 1.06 1.16 =1.2 ,0 , , , , , .75 0.85 0.95 1.06 1.16 1.27 7 ) ) ) ) ) )
[ [ [ [ [ [ > .63 0.63 0.77 0.90 1.03 1.17 1.30 =1.4 ,0 , , , , , .77 0.90 1.03 1.17 1.30 1.44 4 ) ) ) ) ) )
Array B to Array D (min 0.88, max 4.69) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .03 1.03 1.17 1.32 1.47 1.62 1.76 =1.9 ,1 , , , , , .17 1.32 1.47 1.62 1.76 1.91 1 ) ) ) ) ) )
Array C to Array B (min 0.26, max 3.93) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .58 0.58 0.90 1.23 1.55 1.88 2.20 =2.5 ,0 , , , , , .90 1.23 1.55 1.88 2.20 2.53 3 ) ) ) ) ) )
Array D to Array A (min 0.54, max 2.00) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<0
Array B to Array C (min 0.25, max 3.92) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array A (min 0.61, max 3.73) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array A to Array D (min 0.50, max 1.86) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array D (min 0.96, max 5.21) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .33 1.33 1.70 2.08 2.45 2.82 3.19 =3.5 ,1 , , , , , .70 2.08 2.45 2.82 3.19 3.56 6 ) ) ) ) ) )
Array D to Array B (min 0.21, max 1.14) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .31 0.31 0.42 0.52 0.62 0.72 0.82 =0.9 ,0 , , , , , .42 0.52 0.62 0.72 0.82 0.92 2 ) ) ) ) ) )
Array D to Array C (min 0.19, max 1.04) 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<0
[ [ [ [ [ [ > .27 0.27 0.36 0.44 0.52 0.61 0.69 =0.7 ,0 , , , , , .36 0.44 0.52 0.61 0.69 0.77 7 ) ) ) ) ) )
Figure D.4: Each graph shows the probability distribution of the Throughput relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
63
APPENDIX D. FITNESSBUFFERED MODEL TRAINING
Array A to Array B (min 0.48, max 3.16) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .65 0.65 0.83 1.00 1.17 1.34 1.52 =1.6 ,0 , , , , , .83 1.00 1.17 1.34 1.52 1.69 9 ) ) ) ) ) )
Array A to Array C (min 0.57, max 4.75) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .95 0.95 1.32 1.69 2.06 2.44 2.81 =3.1 ,1 , , , , , .32 1.69 2.06 2.44 2.81 3.18 8 ) ) ) ) ) )
Array B to Array A (min 0.32, max 2.08) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02
0.25 0.2 0.15 0.1 0.05
[ [ [ [ [ [ > .48 0.48 0.65 0.81 0.98 1.14 1.31 =1.4 ,0 , , , , , .65 0.81 0.98 1.14 1.31 1.48 8 ) ) ) ) ) )
0
<0
[ [ [ [ [ [ > .63 0.63 1.05 1.46 1.88 2.29 2.71 =3.1 ,1 , , , , , .05 1.46 1.88 2.29 2.71 3.13 3 ) ) ) ) ) )
Array C to Array A (min 0.21, max 1.75) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .36 0.36 0.51 0.66 0.81 0.95 1.10 =1.2 ,0 , , , , , .51 0.66 0.81 0.95 1.10 1.25 5 ) ) ) ) ) )
0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .69 0.69 0.88 1.07 1.27 1.46 1.65 =1.8 ,0 , , , , , .88 1.07 1.27 1.46 1.65 1.84 4 ) ) ) ) ) )
[ [ [ [ [ [ > .41 0.41 0.53 0.65 0.77 0.90 1.02 =1.1 ,0 , , , , , .53 0.65 0.77 0.90 1.02 1.14 4 ) ) ) ) ) )
Array B to Array D (min 0.17, max 1.60) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .27 0.27 0.37 0.47 0.57 0.67 0.77 =0.8 ,0 , , , , , .37 0.47 0.57 0.67 0.77 0.87 7 ) ) ) ) ) )
Array C to Array B (min 0.19, max 4.58) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .33 0.33 0.47 0.61 0.74 0.88 1.02 =1.1 ,0 , , , , , .47 0.61 0.74 0.88 1.02 1.16 6 ) ) ) ) ) )
Array D to Array A (min 0.50, max 3.51) 0.25
<0
Array B to Array C (min 0.22, max 5.24) 0.3
<0
Array A to Array D (min 0.29, max 2.00) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array D (min 0.15, max 1.04) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .23 0.23 0.31 0.39 0.47 0.55 0.63 =0.7 ,0 , , , , , .31 0.39 0.47 0.55 0.63 0.71 1 ) ) ) ) ) )
Array D to Array B (min 0.63, max 5.95) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array D to Array C (min 0.96, max 6.56) 0.25 0.2 0.15 0.1
<0
[ [ [ [ [ [ > .83 0.83 1.04 1.25 1.46 1.66 1.87 =2.0 ,1 , , , , , .04 1.25 1.46 1.66 1.87 2.08 8 ) ) ) ) ) )
0.05
<1
[ [ [ [ [ [ > .38 1.38 1.80 2.22 2.63 3.05 3.47 =3.8 ,1 , , , , , .80 2.22 2.63 3.05 3.47 3.89 9 ) ) ) ) ) )
Figure D.5: Each graph shows the probability distribution of the Latency relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
1
0.9
0.9
0.8
0.8
0.7
0.7
0.5
0.4
Array A Array B Array C Array D
Write size (KB) 1
0.9
0.9
0.8
0.8
0.7
0.7
0.4
400
60
50
40
30
20
10
Write queue
300
0.2 0
0.2
0.3 200
0.3
Array A Array B Array C Array D
Write jump (MB)
Figure D.6: The cumulative distribution of workload characteristics.
800
Array A Array B Array C Array D
100
0.4
0.5
700
0.5
0.6
600
0.6
500
Fraction
1
0
Fraction
Write fraction
150
100
0
0.2 0.7
0.4
0.3
0.2
0.1
0
0.2
Array A Array B Array C Array D
0.3 0.6
0.3
0.5
50
0.4
0.6
250
0.6
200
Fraction
1
0.5
Fraction
64
65
APPENDIX D. FITNESSBUFFERED MODEL TRAINING
Overall Predictor Read queue Write fraction Read jump Read size Write size Write queue Write jump
Score 1.00 0.75 0.51 0.44 0.19 0.17 0.15
Overall Predictor Read queue Write fraction Read jump Read size Write jump Write size Write queue
Score 1.00 0.70 0.56 0.36 0.21 0.20 0.14
Overall Predictor Latency Throughput Bandwidth Read jump Read queue Write fraction Read size Read latency Write size Write jump Write latency
Score 1.00 0.92 0.74 0.31 0.24 0.19 0.15 0.11 0.08 0.06 0.05
Overall Predictor Bandwidth Read latency Latency Read jump Read queue Write fraction Read size Throughput Write jump Write size Write queue
Score 1.00 0.96 0.95 0.80 0.77 0.64 0.57 0.44 0.26 0.25 0.22
Absolute Bandwidth Throughput Predictor Score Predictor Score Read queue 1.00 Read queue 1.00 Read jump 0.61 Read jump 0.67 Read size 0.52 Read size 0.55 Write size 0.33 Write queue 0.22 Write fraction 0.26 Write jump 0.17 Write jump 0.20 Write fraction 0.13 Write queue 0.13 Write size 0.10 Relative Bandwidth Throughput Predictor Score Predictor Score Read queue 1.00 Read queue 1.00 Read jump 0.79 Read jump 0.73 Read size 0.36 Read size 0.34 Write jump 0.32 Write queue 0.25 Write fraction 0.30 Write size 0.19 Write size 0.23 Write jump 0.17 Write queue 0.05 Write fraction 0.16 Relative Performance Bandwidth Throughput Predictor Score Predictor Score Bandwidth 1.00 Throughput 1.00 Read jump 0.27 Read queue 0.17 Read latency 0.14 Read jump 0.10 Read queue 0.10 Read size 0.10 Write size 0.07 Latency 0.07 Write fraction 0.05 Write fraction 0.06 Throughput 0.04 Write latency 0.04 Read size 0.04 Bandwidth 0.03 Write jump 0.04 Write queue 0.03 Latency 0.04 Write size 0.02 Write latency 0.01 Write jump 0.02 Relative Fitness Bandwidth Throughput Predictor Score Predictor Score Read latency 1.00 Read latency 1.00 Read queue 0.92 Read jump 0.76 Read jump 0.90 Bandwidth 0.70 Bandwidth 0.80 Read queue 0.66 Write fraction 0.61 Write fraction 0.61 Read size 0.48 Latency 0.59 Latency 0.39 Read size 0.46 Throughput 0.37 Throughput 0.37 Write queue 0.24 Write size 0.36 Write size 0.12 Write queue 0.27 Write jump 0.09 Write jump 0.11
Latency Predictor Write fraction Read queue Read jump Read size Write queue Write size Write jump
Score 1.00 0.53 0.14 0.13 0.09 0.06 0.04
Latency Predictor Write fraction Read queue Read size Read jump Write size Write jump Write queue
Score 1.00 0.69 0.26 0.17 0.13 0.09 0.09
Latency Predictor Latency Write fraction Bandwidth Read size Read jump Read queue Read latency Write jump Write latency Throughput Write queue
Score 1.00 0.12 0.06 0.05 0.05 0.03 0.02 0.02 0.02 0.02 0.02
Latency Predictor Latency Bandwidth Read size Write jump Read latency Write fraction Throughput Read queue Read jump Write size Write queue
Score 1.00 0.72 0.34 0.33 0.26 0.25 0.24 0.23 0.22 0.09 0.02
Table D.4: The normalized importance measure of each predictor.
Model Absolute Relative Relative Performance Relative Fitness
Overall 13.5 14.3 12.7 8.7
Absolute Array Array Array Array Relative Array Array Array Array Relative Performance Array Array Array Array Relative Fitness Array Array Array Array
Absolute Array A Array B Array C Array D
A 19.0 -
Relative Array A Array B Array C Array D
A 19.0 17.0 14.0 20.0
Relative Performance Array A Array B Array C Array D
A 15.0 6.0 20.0 14.0
Relative Fitness Array A Array B Array C Array D
A 1 8.0 17.0 16.0
Bandwidth B C 13.0 8.0 Bandwidth B C 10.0 6.0 13.0 9.0 24.0 8.0 20.0 11.0 Bandwidth B C 8.0 15.0 17.0 6.0 21.0 16.0 11.0 4.0 Bandwidth B C 7.0 15.0 1 1.0 18.0 1 6.0 2.0
Bandwidth 15.0 14.0 12.2 9.4
Pairwise Array 19.7 Array 19.7 15.7 18.0 15.0 Array 16.3 12.3 18.3 16.3 Array 1 13.3 11.3 12.7
Array 14.3 Array 13.3 14.3 18.3 14.3 Array 10.3 17.3 20.0 12.7 Array 10.7 1 11.0 6.0
D 20.0
A 18.0 -
D 7.0 18.0 12.0 20.0
A 18.0 10.0 17.0 3.0
D 11.0 11.0 20.0 17.0
A 17.0 18.0 12.0 17.0
D 5.0 8.0 10.0 1
A 1 17.0 14.0 13.0
Throughput 10.2 9.9 13.0 10.9
Array 8.0 Array 9.3 7.3 8.0 11.0 Array 15.7 6.7 16.3 4.3 Array 9.7 7.0 1 1.7
Throughput B C 14.0 8.0 Throughput B C 10.0 1.0 14.0 1.0 20.0 8.0 4.0 1.0 Throughput B C 12.0 18.0 16.0 7.0 17.0 17.0 14.0 3.0 Throughput B C 11.0 13.0 1 19.0 11.0 1 8.0 2.0
Latency 15.2 19.0 13.0 5.8
Array 12.0 Array 15.3 17.7 16.3 12.0 Array 15.0 9.3 12.0 15.0 Array 6.7 7.3 7.3 1
D 1.0
A 22.0 -
D 16.0 21.0 15.0 1.0
A 22.0 20.0 23.0 22.0
D 18.0 8.0 12.0 16.0
A 17.0 13.0 23.0 18.0
D 5.0 9.0 9.0 1
A 1 15.0 3.0 9.0
Table D.5: Tree sizes (leaf nodes) and their averages.
Latency B C 16.0 8.0 Latency B C 20.0 21.0 16.0 12.0 11.0 8.0 19.0 21.0 Latency B C 11.0 14.0 19.0 7.0 22.0 16.0 13.0 6.0 Latency B C 14.0 1.0 1 1.0 4.0 1 4.0 1.0
D 15.0 D 23.0 14.0 22.0 15.0 D 16.0 9.0 4.0 12.0 D 10.0 5.0 3.0 1
67
APPENDIX D. FITNESSBUFFERED MODEL TRAINING
Bandwidth error (FitnessBuffered) Median relative error (Avg.)
Median relative error (Avg.)
Overall error (FitnessBuffered) 0.26 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08
70
30 AM (i=j) AM
RF
0.28 0.26 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
RM RM’
70 RF
Latency error (FitnessBuffered) Median relative error (Avg.)
Median relative error (Avg.)
Throughput error (FitnessBuffered)
40 50 60 Samples used for training (%)
70 RF
0.3 0.28 0.26 0.24 0.22 0.2 0.18 0.16 0.14 0.12 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
Figure D.7: Error vs. training. The training set size varies from 25% of the collected samples to 75%; the remaining samples are used for testing. For each training set size, the Median relative error is calculated for each pairing of devices. They are then averaged to produce the values shown.
68
Bandwidth error (FitnessBuffered) Mean relative error (Avg.)
Mean relative error (Avg.)
Overall error (FitnessBuffered) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 30
40 50 60 Samples used for training (%) RM RM’
AM (i=j) AM
0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1
70
30
RF
AM (i=j) AM
0.45 0.4 0.35 0.3 0.25 0.2 0.15 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
RM RM’
70 RF
Latency error (FitnessBuffered)
0.5
Mean relative error (Avg.)
Mean relative error (Avg.)
Throughput error (FitnessBuffered)
40 50 60 Samples used for training (%)
70 RF
0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
Figure D.8: Error vs. training. The training set size varies from 25% of the collected samples to 75%; the remaining samples are used for testing. For each training set size, the Mean relative error is calculated for each pairing of devices. They are then averaged to produce the values shown.
Appendix E
FitnessFS model training
Application FS (fs) Total used
Samples 200
Iters 3
First sample 0 100
Last sample 99
Table E.1: Multiple training samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Array A 0.30 104.56 59.72 71.10 630.12 25.12 2.87 23.48 302.07 24.69 51.24 9.84
Array B 0.30 106.12 59.59 55.13 629.82 20.08 2.83 17.59 227.98 21.51 39.88 15.58
Array C 0.30 109.11 59.30 39.51 636.29 21.22 2.91 15.43 203.18 49.03 116.70 16.99
Array D 0.30 105.15 59.10 110.04 638.64 20.34 2.83 23.88 316.14 17.55 23.68 13.15
MAD 0.0 1.4 0.2 21.6 3.8 1.7 0.0 3.6 46.8 10.4 29.4 2.4
COV 0.6% 1.6% 0.4% 38.0% 0.6% 9.3% 1.1% 18.3% 18.2% 43.6% 61.1% 19.5%
Max. Diff. 0.0% 4.3% 1.1% 178.5% 1.4% 25.1% 2.8% 54.8% 55.6% 179.4% 392.8% 72.7%
Table E.2: Workload characteristics and performance are measured for each sample, on each storage device. The average value for each measurement is reported in this table. The mean absolute deviation (MAD), coefficient of variation (COV), and maximum relative differences of these averages are also reported; these metrics quantify how the averages change among the storage devices. For example, the mean average deviation of the averages for Read latency (ms) is 2.40, their coeefficient of variation is 19.547%, and the maximum relative difference is 72.66%.
69
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 0.8% 0.3% 0.1% 7.9% 0.9% 4.6% 1.2% 1.1% 1.0% 0.7% 3.4% 0.2%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 1.0% 0.3% 0.1% 5.1% 0.1% 2.8% 1.7% 1.9% 1.3% 2.9% 6.8% 3.0%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 2.7% 0.9% 0.0% 2.7% 0.7% 7.1% 0.5% 1.3% 1.1% 4.3% 7.9% 0.1%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 1.9% 0.1% 0.3% 51.4% 1.3% 2.7% 1.2% 2.3% 1.7% 0.6% 1.4% 10.2%
Array A Min 25% 0.0 0.1 0.0 7.0 0.0 28.8 0.0 3.0 0.0 427.0 0.0 16.2 0.0 1.1 5.3 16.3 135.0 210.0 1.0 14.7 0.0 43.1 0.0 5.5 Array B Min 25% 0.0 0.1 0.0 7.6 0.0 28.6 0.0 3.0 0.0 416.0 0.0 10.1 0.0 1.1 3.5 12.3 65.0 160.0 1.2 12.6 0.0 22.1 0.0 10.1 Array C Min 25% 0.0 0.1 0.0 6.8 0.0 28.1 0.0 2.0 0.0 407.0 0.0 11.6 0.0 1.2 3.6 9.6 74.0 124.0 2.1 27.9 0.0 88.6 0.0 7.8 Array D Min 25% 0.0 0.1 0.0 6.5 4.0 28.8 0.0 3.0 0.0 428.0 0.0 10.9 1.0 1.1 4.6 13.4 106.0 172.0 0.9 9.9 0.0 4.4 0.2 9.1
50% 0.3 124.3 59.3 70.0 635.0 20.6 1.6 20.7 252.0 22.8 54.9 7.6
75% 0.4 174.4 87.2 104.0 854.0 31.8 3.1 25.0 303.0 27.8 63.6 11.4
90% 0.6 191.1 113.3 132.0 1005.0 53.5 5.3 45.4 454.0 38.5 80.5 18.9
Max 1.0 240.9 237.1 765.0 1446.0 69.0 12.7 65.3 2187.0 105.1 134.1 46.2
Mean 0.3 104.6 59.7 71.1 630.1 25.1 2.9 23.5 302.1 24.7 51.2 9.8
50% 0.3 125.8 59.3 51.0 631.0 15.0 1.6 15.6 194.0 17.7 32.5 11.7
75% 0.4 176.7 87.7 92.0 870.0 24.8 3.1 20.2 238.0 22.1 41.0 15.2
90% 0.6 192.6 112.0 123.0 1023.0 53.4 5.4 28.1 341.0 28.5 67.0 22.7
Max 1.0 240.6 240.7 186.0 1445.0 64.0 11.9 51.2 1472.0 192.5 453.3 93.5
Mean 0.3 106.1 59.6 55.1 629.8 20.1 2.8 17.6 228.0 21.5 39.9 15.6
50% 0.3 129.8 59.1 20.0 645.0 16.0 1.6 12.1 144.0 44.4 115.9 11.2
75% 0.4 181.8 88.2 65.0 848.0 31.0 3.4 15.3 187.0 53.4 132.3 16.0
90% 0.6 201.3 113.7 107.0 1030.0 53.0 6.0 21.6 269.0 80.2 205.5 37.7
Max 1.0 242.8 170.6 203.0 1449.0 64.0 12.0 86.2 2294.0 249.6 399.3 121.5
Mean 0.3 109.1 59.3 39.5 636.3 21.2 2.9 15.4 203.2 49.0 116.7 17.0
50% 0.3 124.8 59.2 65.0 625.0 15.6 1.6 18.0 225.0 14.3 23.7 10.7
75% 0.4 177.2 87.7 103.0 865.0 25.0 3.1 26.2 320.0 16.9 29.0 12.9
90% 0.6 192.8 113.2 124.0 1030.0 52.0 5.6 42.2 507.0 22.7 36.8 18.8
Max 1.0 238.2 155.1 4144.0 1476.0 64.0 12.3 100.7 3339.0 168.6 170.8 139.7
Mean 0.3 105.2 59.1 110.0 638.6 20.3 2.8 23.9 316.1 17.6 23.7 13.1
Table E.3: Workload characteristics and performance are measured for each sample, on each storage device. The minimum value, percentiles, maximum value and average are reported for each measurement. In addition, the relative difference between the average performance of the best and second-to-best iteration is reported. This value quantifies the change in a given measurement across multiple runs of the same sample on the same storage device.
71
APPENDIX E. FITNESSFS MODEL TRAINING 1
Bandwidth (MB/sec)
0.6 0.4
40
30
20
0
10
0
Array A Array B Array C Array D
0
60
Array A Array B Array C Array D
0.2
50
Fraction
0.8
110 100 90 80 70 60 50 40 30 20 10 0
10
20
30
Bandwidth (MB/sec) 1
3500
0.9 Throughput (IO/sec)
0.6 0.5 0.4 0.3
Array A Array B Array C Array D 500
400
300
200
100
0
0
2500
1500 1000 500 0 0
Throughput (IO/sec) 1
250
0.8
200 Latency (ms)
0.6 0.5 0.4
60
50
40
30
20
0
10
0
Latency (ms)
150 100 50
100
0.1
90
0.2
80
Array A Array B Array C Array D 70
Fraction
0.7
10 20 30 40 50 60 70 80 90 100 Sample number Array A Array B Array C Array D
0.9
0.3
90 100
2000
800
0.1
700
0.2
600
Fraction
0.7
80
Array A Array B Array C Array D
3000
0.8
40 50 60 70 Sample number
0 0
10
20
30
40 50 60 70 Sample number
80
90 100
Figure E.1: The cumulative distribution of performance is shown (top). In addition, the performance of each array is shown (bottom), sorted by the performance of Array C.
72
Bandwidth (MB/sec) on Array A
Throughput (IO/sec) on Array A
0.3
0.3
0.25
0.25
0.25
0.2 0.15
0.2 0.15 0.1
0.1
0
Fraction
0.3
0.05
0.05
0
<1
1
[11
[17 [22 [28 [34 [40 >= 4 , , , , , 7) 22) 28) 34) 40) 45) 5
<1
[ [ [ [ [ [ > 86 186, 237, 288, 338, 389, 440, =49 1 23 2 3 3 4 4 7) 88) 38) 89) 40) 91)
Bandwidth (MB/sec) on Array B
0
0.3
0.25
0.25 0.2 0.15 0.1 0.05
0.05 0
Fraction
0.3 0.25 0.2
0
<8
[8,
13
)
[13
[17 [22 [26 [31 >= 3 , , , , 7) 22) 26) 31) 36) 6
<1
[ [ [ [ [ [ > 08 108, 152, 196, 239, 282, 326, =37 0 15 1 2 2 3 3 2) 96) 39) 82) 26) 70)
Fraction
0.1 0.05 0
<7
[7,
10
)
[10
0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
[14 [17 [20 [23 >= 2 , , , , 4) 17) 20) 23) 27) 7
[ [ [ [ [ [ > 18 118, 162, 205, 249, 293, 336, =38 0 16 2 2 2 3 3 2) 05) 49) 93) 36) 80)
Bandwidth (MB/sec) on Array D
0.1
0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
0.25
0
Fraction
0.3 0.25 Fraction
0.3
0.2 0.15 0.1 0.05
0.05
0
<1
1
[11
[18 [24 [30 [37 [43 >= 5 , , , , , 8) 24) 30) 37) 43) 50) 0
,1
13
)
[13
[18 [24 [30 [35 >= 4 , , , , 8) 24) 30) 35) 41) 1
,1
<1
[14
[25 [36 [48 [59 [71 >= 8 , , , , , 5) 36) 48) 59) 71) 82) 2
,2
Latency (ms) on Array D
0.25
0.1
[7,
Throughput (IO/sec) on Array D 0.3
0.15
<7
4
0.35
0.2
[20 [27 [33 [40 >= 4 , , , , 0) 27) 33) 40) 46) 6
,2
Latency (ms) on Array C
<1
,1
0.35
[14
0.2
Throughput (IO/sec) on Array C
0.3
0.2
)
0.15
0
Fraction
Bandwidth (MB/sec) on Array C 0.35
0.15
14
0.05
,1
0.25
[7,
Latency (ms) on Array B 0.3
0.15
<7
Throughput (IO/sec) on Array B
Fraction
Fraction
0.1
0.35
0.1
Fraction
0.2 0.15
0.05
,1
0.35
Fraction
Latency (ms) on Array A
0.35
Fraction
Fraction
0.35
0.2 0.15 0.1 0.05
<1
[ [ [ [ [ [ > 87 187, 268, 348, 429, 510, 590, =67 1 26 3 4 5 5 6 8) 48) 29) 10) 90) 71)
0
<4
Figure E.2: Probability distributions of performance.
[4,
8)
[8,
11
)
[11
[14 [18 [21 >= 2 , , , 4) 18) 21) 24) 4
,1
73
APPENDIX E. FITNESSFS MODEL TRAINING
Array A to Array B (min 0.32, max 1.13)
Array A to Array C (min 0.31, max 1.47)
Array A to Array D (min 0.59, max 1.77)
0.3
0.3
0.3
0.25
0.25
0.25
0.2
0.2
0.2
0.15
0.15
0.15
0.1
0.1
0.1
0.05
0.05
0.05
0
<0
[ [ [ [ [ [ > .40 0.40 0.49 0.58 0.67 0.76 0.85 =0.9 ,0 , , , , , .49 0.58 0.67 0.76 0.85 0.93 3 ) ) ) ) ) )
0
<0
[ [ [ [ [ [ > .40 0.40 0.49 0.59 0.68 0.77 0.87 =0.9 ,0 , , , , , .49 0.59 0.68 0.77 0.87 0.96 6 ) ) ) ) ) )
Array B to Array A (min 0.88, max 3.17) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .04 1.04 1.20 1.36 1.52 1.68 1.84 =2.0 ,1 , , , , , .20 1.36 1.52 1.68 1.84 2.00 0 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .93 0.93 1.19 1.44 1.69 1.95 2.20 =2.4 ,1 , , , , , .19 1.44 1.69 1.95 2.20 2.45 5 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .48 0.48 0.64 0.80 0.96 1.13 1.29 =1.4 ,0 , , , , , .64 0.80 0.96 1.13 1.29 1.45 5 ) ) ) ) ) )
<0
.67
[0.
[ [ [ [ [ > 67 0.77 0.87 0.97 1.07 1.17 =1.2 ,0 , , , , , .77 0.87 0.97 1.07 1.17 1.27 7 ) ) ) ) ) )
[ [ [ [ [ [ > .70 0.70 0.81 0.91 1.02 1.13 1.24 =1.3 ,0 , , , , , .81 0.91 1.02 1.13 1.24 1.34 4 ) ) ) ) ) )
Array B to Array D (min 0.93, max 5.60) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .09 1.09 1.26 1.42 1.59 1.76 1.92 =2.0 ,1 , , , , , .26 1.42 1.59 1.76 1.92 2.09 9 ) ) ) ) ) )
Array C to Array B (min 0.23, max 3.19) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .48 0.48 0.73 0.98 1.23 1.48 1.73 =1.9 ,0 , , , , , .73 0.98 1.23 1.48 1.73 1.98 8 ) ) ) ) ) )
Array D to Array A (min 0.57, max 1.68) 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<0
Array B to Array C (min 0.31, max 4.43) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array A (min 0.68, max 3.24) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0
Array C to Array D (min 0.98, max 4.25) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .25 1.25 1.52 1.79 2.07 2.34 2.61 =2.8 ,1 , , , , , .52 1.79 2.07 2.34 2.61 2.89 9 ) ) ) ) ) )
Array D to Array B (min 0.18, max 1.08) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array D to Array C (min 0.24, max 1.02) 0.35 0.3 0.25 0.2 0.15 0.1
<0
.28
[0.
[ [ [ [ [ > 28 0.39 0.50 0.60 0.71 0.81 =0.9 ,0 , , , , , .39 0.50 0.60 0.71 0.81 0.92 2 ) ) ) ) ) )
0.05
<0
[ [ [ [ [ [ > .32 0.32 0.40 0.49 0.57 0.66 0.74 =0.8 ,0 , , , , , .40 0.49 0.57 0.66 0.74 0.83 3 ) ) ) ) ) )
Figure E.3: Each graph shows the probability distribution of the Bandwidth relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
74
Array A to Array B (min 0.30, max 1.13) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .39 0.39 0.48 0.57 0.66 0.75 0.84 =0.9 ,0 , , , , , .48 0.57 0.66 0.75 0.84 0.93 3 ) ) ) ) ) )
Array A to Array C (min 0.30, max 1.98) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.2 0.15 0.1 0.05
<0
[ [ [ [ [ [ > .39 0.39 0.49 0.58 0.67 0.77 0.86 =0.9 ,0 , , , , , .49 0.58 0.67 0.77 0.86 0.96 6 ) ) ) ) ) )
Array B to Array A (min 0.88, max 3.29) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .06 1.06 1.24 1.42 1.59 1.77 1.95 =2.1 ,1 , , , , , .24 1.42 1.59 1.77 1.95 2.13 3 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .78 0.78 1.05 1.32 1.60 1.87 2.14 =2.4 ,1 , , , , , .05 1.32 1.60 1.87 2.14 2.41 1 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .46 0.46 0.62 0.78 0.93 1.09 1.25 =1.4 ,0 , , , , , .62 0.78 0.93 1.09 1.25 1.40 0 ) ) ) ) ) )
0.25 0.2 0.15 0.1 0.05 0
<0
.47
[0.
[ [ [ [ [ > 47 0.60 0.73 0.86 0.99 1.12 =1.2 ,0 , , , , , .60 0.73 0.86 0.99 1.12 1.25 5 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .71 0.71 0.81 0.92 1.02 1.13 1.24 =1.3 ,0 , , , , , .81 0.92 1.02 1.13 1.24 1.34 4 ) ) ) ) ) )
Array B to Array D (min 0.94, max 9.60) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .09 1.09 1.25 1.40 1.55 1.71 1.86 =2.0 ,1 , , , , , .25 1.40 1.55 1.71 1.86 2.01 1 ) ) ) ) ) )
Array C to Array B (min 0.15, max 3.26) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .41 0.41 0.67 0.93 1.19 1.45 1.71 =1.9 ,0 , , , , , .67 0.93 1.19 1.45 1.71 1.97 7 ) ) ) ) ) )
Array D to Array A (min 0.34, max 1.67) 0.3
0
Array B to Array C (min 0.31, max 6.52) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array A (min 0.50, max 3.34) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array A to Array D (min 0.60, max 2.92) 0.25
Array C to Array D (min 0.98, max 4.33) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .25 1.25 1.52 1.79 2.06 2.33 2.60 =2.8 ,1 , , , , , .52 1.79 2.06 2.33 2.60 2.87 7 ) ) ) ) ) )
Array D to Array B (min 0.10, max 1.06) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array D to Array C (min 0.23, max 1.02) 0.3 0.25 0.2 0.15 0.1
<0
.22
[0.
[ [ [ [ [ > 22 0.34 0.45 0.57 0.68 0.80 =0.9 ,0 , , , , , .34 0.45 0.57 0.68 0.80 0.91 1 ) ) ) ) ) )
0.05
<0
[ [ [ [ [ [ > .32 0.32 0.40 0.48 0.57 0.65 0.74 =0.8 ,0 , , , , , .40 0.48 0.57 0.65 0.74 0.82 2 ) ) ) ) ) )
Figure E.4: Each graph shows the probability distribution of the Throughput relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
75
APPENDIX E. FITNESSFS MODEL TRAINING
Array A to Array B (min 0.37, max 3.87) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .54 0.54 0.70 0.86 1.03 1.19 1.35 =1.5 ,0 , , , , , .70 0.86 1.03 1.19 1.35 1.52 2 ) ) ) ) ) )
Array A to Array C (min 0.53, max 5.25) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .90 0.90 1.26 1.63 1.99 2.36 2.73 =3.0 ,1 , , , , , .26 1.63 1.99 2.36 2.73 3.09 9 ) ) ) ) ) )
Array B to Array A (min 0.26, max 2.69) 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .44 0.44 0.63 0.81 1.00 1.18 1.37 =1.5 ,0 , , , , , .63 0.81 1.00 1.18 1.37 1.55 5 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .32 0.32 0.45 0.58 0.71 0.84 0.97 =1.1 ,0 , , , , , .45 0.58 0.71 0.84 0.97 1.10 0 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .68 0.68 1.17 1.66 2.14 2.63 3.12 =3.6 ,1 , , , , , .17 1.66 2.14 2.63 3.12 3.61 1 ) ) ) ) ) )
[ [ [ [ [ [ > .37 0.37 0.48 0.59 0.71 0.82 0.93 =1.0 ,0 , , , , , .48 0.59 0.71 0.82 0.93 1.04 4 ) ) ) ) ) )
Array B to Array D (min 0.14, max 1.65) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .25 0.25 0.35 0.46 0.57 0.67 0.78 =0.8 ,0 , , , , , .35 0.46 0.57 0.67 0.78 0.89 9 ) ) ) ) ) )
Array C to Array B (min 0.16, max 5.31) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .28 0.28 0.40 0.52 0.64 0.76 0.88 =1.0 ,0 , , , , , .40 0.52 0.64 0.76 0.88 1.00 0 ) ) ) ) ) )
Array D to Array A (min 0.62, max 3.88) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
Array B to Array C (min 0.19, max 6.17) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
Array C to Array A (min 0.19, max 1.88) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array A to Array D (min 0.26, max 1.60) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array D (min 0.11, max 0.97) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .19 0.19 0.27 0.35 0.44 0.52 0.60 =0.6 ,0 , , , , , .27 0.35 0.44 0.52 0.60 0.68 8 ) ) ) ) ) )
Array D to Array B (min 0.61, max 7.10)
Array D to Array C (min 1.03, max 9.41)
0.6
0.3
0.5
0.25
0.4
0.2
0.3 0.15
0.2
0.1
0.1
<0
.84
[0.
[ [ [ [ [ > 84 1.05 1.26 1.48 1.69 1.90 =2.1 ,1 , , , , , .05 1.26 1.48 1.69 1.90 2.12 2 ) ) ) ) ) )
0
<0
.83
[0.
[ [ [ [ [ > 83 1.05 1.27 1.49 1.71 1.93 =2.1 ,1 , , , , , .05 1.27 1.49 1.71 1.93 2.15 5 ) ) ) ) ) )
0.05
<1
[ [ [ [ [ [ > .52 1.52 2.00 2.48 2.96 3.44 3.93 =4.4 ,2 , , , , , .00 2.48 2.96 3.44 3.93 4.41 1 ) ) ) ) ) )
Figure E.5: Each graph shows the probability distribution of the Latency relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
76
1
1
0.9
0.9 0.8 0.7
0.5
Array A Array B Array C Array D
0.2 0.1
Write fraction
150
100
0.8
0
0 0.7
0.5
0.4
0.3
0.2
0.1
0
0.2
0.4 0.3
Array A Array B Array C Array D
0.3
0.5
50
0.4
0.6
250
0.6
200
Fraction
0.7
0.6
Fraction
0.8
Write size (KB)
1
1
0.9
0.9
0.8
0.8
0.4
Array A Array B Array C Array D
100
80
70
60
50
40
30
10
20
Write queue
60
0.2 0
0
0.3 40
0.1
Array A Array B Array C Array D 20
0.2
0.5
Write jump (MB)
Figure E.6: The cumulative distribution of workload characteristics.
180
0.3
160
0.4
0.6
140
0.5
0.7
120
Fraction
0.6
0
Fraction
0.7
77
APPENDIX E. FITNESSFS MODEL TRAINING
Overall Predictor Write fraction Read queue Read jump Write size Write queue Read size
Score 1.00 0.92 0.53 0.35 0.18 0.18
Overall Predictor Write fraction Read queue Read jump Write size Write jump Read size Write queue
Score 1.00 0.89 0.63 0.42 0.31 0.30 0.20
Overall Predictor Latency Bandwidth Throughput Read queue Write size Read jump Write fraction Read size Read latency Write jump Write queue Write latency
Score 1.00 0.88 0.76 0.29 0.28 0.26 0.20 0.18 0.11 0.08 0.07 0.07
Overall Predictor Bandwidth Read latency Read queue Read jump Latency Read size Throughput Write fraction Write latency Write size Write queue
Score 1.00 0.88 0.84 0.80 0.66 0.65 0.56 0.55 0.33 0.33 0.29
Absolute Bandwidth Throughput Predictor Score Predictor Score Read queue 1.00 Read queue 1.00 Write fraction 0.88 Write size 0.81 Read jump 0.79 Read jump 0.60 Write jump 0.51 Read size 0.31 Write queue 0.34 Write queue 0.16 Write size 0.19 Write fraction 0.10 Relative Bandwidth Throughput Predictor Score Predictor Score Read queue 1.00 Read queue 1.00 Write fraction 0.79 Read jump 0.77 Read jump 0.74 Write size 0.77 Write jump 0.61 Read size 0.38 Read size 0.35 Write queue 0.29 Write size 0.33 Write fraction 0.24 Write queue 0.14 Write jump 0.09 Relative Performance Bandwidth Throughput Predictor Score Predictor Score Bandwidth 1.00 Throughput 1.00 Read queue 0.14 Write size 0.27 Read jump 0.12 Read queue 0.21 Latency 0.09 Read size 0.14 Write fraction 0.09 Read jump 0.12 Read latency 0.07 Latency 0.10 Write jump 0.06 Write fraction 0.07 Write size 0.05 Read latency 0.04 Read size 0.03 Bandwidth 0.04 Throughput 0.03 Write queue 0.01 Write queue 0.02 Write latency 0.01 Write latency 0.00 Write jump 0.00 Relative Fitness Bandwidth Throughput Predictor Score Predictor Score Read queue 1.00 Read latency 1.00 Read latency 0.92 Read jump 0.90 Read jump 0.87 Bandwidth 0.80 Bandwidth 0.65 Throughput 0.64 Throughput 0.57 Read queue 0.61 Latency 0.51 Read size 0.58 Write fraction 0.32 Write fraction 0.41 Write queue 0.31 Latency 0.39 Read size 0.24 Write latency 0.23 Write size 0.23 Write size 0.22 Write latency 0.17 Write jump 0.19
Latency Predictor Write fraction Read queue Read jump Write queue Write size Read size
Score 1.00 0.40 0.11 0.03 0.03 0.02
Latency Predictor Write fraction Read queue Read jump Write jump Write size Read size Write queue
Score 1.00 0.37 0.20 0.11 0.11 0.10 0.10
Latency Predictor Latency Write fraction Read jump Write latency Read size Bandwidth Write queue Write size Write jump Read queue Throughput Read latency
Score 1.00 0.10 0.08 0.07 0.06 0.05 0.05 0.05 0.03 0.03 0.03 0.02
Latency Predictor Bandwidth Read size Latency Write fraction Write jump Read queue Write latency Write size Write queue Read latency Read jump
Score 1.00 0.77 0.72 0.61 0.44 0.43 0.42 0.34 0.28 0.23 0.18
Table E.4: The normalized importance measure of each predictor.
Model Absolute Relative Relative Performance Relative Fitness
Overall 10.9 13.2 13.2 7.7
Absolute Array Array Array Array Relative Array Array Array Array Relative Performance Array Array Array Array Relative Fitness Array Array Array Array
Absolute Array A Array B Array C Array D
A 16.0 -
Relative Array A Array B Array C Array D
A 16.0 16.0 22.0 21.0
Relative Performance Array A Array B Array C Array D
A 17.0 11.0 11.0 19.0
Relative Fitness Array A Array B Array C Array D
A 1 3.0 18.0 9.0
Bandwidth B C 18.0 4.0 Bandwidth B C 19.0 16.0 18.0 8.0 15.0 4.0 18.0 4.0 Bandwidth B C 21.0 13.0 17.0 18.0 8.0 18.0 15.0 16.0 Bandwidth B C 11.0 8.0 1 3.0 18.0 1 3.0 2.0
Bandwidth 13.2 15.9 14.8 8.6
Pairwise Array 12.0 Array 12.0 13.7 17.3 14.0 Array 16.7 11.0 12.0 18.7 Array 1 5.3 10.7 10.7
Array 10.3 Array 16.0 10.3 15.7 9.7 Array 18.0 17.0 8.0 13.7 Array 7.3 1 13.3 3.7
D 15.0
A 15.0 -
D 21.0 20.0 11.0 15.0
A 15.0 17.0 15.0 10.0
D 18.0 18.0 9.0 16.0
A 17.0 15.0 15.0 19.0
D 17.0 4.0 7.0 1
A 1 8.0 5.0 17.0
Throughput 8.5 10.0 12.8 7.9
Array 11.7 Array 11.3 8.7 11.7 7.7 Array 7.7 16.7 17.0 14.0 Array 7.0 4.3 1 2.3
Throughput B C 5.0 13.0 Throughput B C 21.0 1.0 5.0 1.0 13.0 13.0 2.0 1.0 Throughput B C 23.0 1.0 17.0 16.0 11.0 16.0 9.0 7.0 Throughput B C 9.0 3.0 1 5.0 18.0 1 5.0 4.0
Latency 11.0 13.8 11.9 6.5
Array 9.7 Array 15.7 16.0 13.0 9.7 Array 14.0 14.7 9.7 12.3 Array 16.0 6.3 5.0 1
D 1.0
A 5.0 -
D 12.0 13.0 14.0 1.0
A 5.0 8.0 15.0 11.0
D 13.0 11.0 14.0 2.0
A 16.0 7.0 10.0 18.0
D 12.0 3.0 6.0 1
A 1 5.0 9.0 6.0
Table E.5: Tree sizes (leaf nodes) and their averages.
Latency B C 8.0 18.0 Latency B C 8.0 17.0 8.0 17.0 19.0 18.0 9.0 18.0 Latency B C 10.0 9.0 17.0 16.0 5.0 17.0 17.0 19.0 Latency B C 2.0 10.0 1 5.0 4.0 1 3.0 1.0
D 13.0 D 14.0 15.0 14.0 13.0 D 11.0 15.0 6.0 19.0 D 19.0 12.0 2.0 1
79
APPENDIX E. FITNESSFS MODEL TRAINING
Bandwidth error (FitnessFS) Median relative error (Avg.)
Median relative error (Avg.)
Overall error (FitnessFS) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08
70
30 AM (i=j) AM
RF
0.18 0.17 0.16 0.15 0.14 0.13 0.12 0.11 0.1 0.09 0.08 0.07 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
RM RM’
70 RF
Latency error (FitnessFS) Median relative error (Avg.)
Median relative error (Avg.)
Throughput error (FitnessFS)
40 50 60 Samples used for training (%)
70 RF
0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
Figure E.7: Error vs. training. The training set size varies from 25% of the collected samples to 75%; the remaining samples are used for testing. For each training set size, the Median relative error is calculated for each pairing of devices. They are then averaged to produce the values shown.
80
Bandwidth error (FitnessFS) Mean relative error (Avg.)
Mean relative error (Avg.)
Overall error (FitnessFS) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1
70
30
RF
AM (i=j) AM
30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
RM RM’
70 RF
Latency error (FitnessFS)
0.26 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1
Mean relative error (Avg.)
Mean relative error (Avg.)
Throughput error (FitnessFS)
40 50 60 Samples used for training (%)
70 RF
0.55 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
Figure E.8: Error vs. training. The training set size varies from 25% of the collected samples to 75%; the remaining samples are used for testing. For each training set size, the Mean relative error is calculated for each pairing of devices. They are then averaged to produce the values shown.
Appendix F
FitnessCache model training
Application Cache (cache) Total used
Samples 70
Iters 3
First sample 0 35
Last sample 34
Table F.1: Multiple training samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Array A 0.00 0.00 15.77 0.00 1855.03 0.00 6.76 16.18 1473.03 6.71 0.00 6.71
Array B 0.00 0.00 15.77 0.00 1861.46 0.00 6.68 15.66 1020.89 7.89 0.00 7.89
Array C 0.00 0.00 15.77 0.00 1856.29 0.00 5.70 35.25 5534.83 14.40 0.00 14.40
Array D 0.00 0.00 15.77 0.00 1850.03 0.00 5.59 40.11 6542.57 4.46 0.00 4.46
MAD 0.0 0.0 0.0 0.0 3.2 0.0 0.5 10.9 2395.9 3.0 0.0 3.0
COV nan% nan% 0.0% nan% 0.2% nan% 8.7% 41.1% 66.6% 44.2% nan% 44.2%
Max. Diff. nan% nan% 0.0% nan% 0.6% nan% 20.9% 156.1% 540.9% 222.9% nan% 222.9%
Table F.2: Workload characteristics and performance are measured for each sample, on each storage device. The average value for each measurement is reported in this table. The mean absolute deviation (MAD), coefficient of variation (COV), and maximum relative differences of these averages are also reported; these metrics quantify how the averages change among the storage devices. For example, the mean average deviation of the averages for Read latency (ms) is 3.02, their coeefficient of variation is 44.193%, and the maximum relative difference is 222.87%.
81
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. nan% nan% 0.0% nan% 0.4% nan% 0.3% 0.3% 1.0% 0.1% nan% 0.1%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. nan% nan% 0.0% nan% 0.1% nan% 0.0% 0.0% 1.0% 0.2% nan% 0.2%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. nan% nan% 0.0% nan% 0.3% nan% 0.2% 0.0% 0.3% 0.0% nan% 0.1%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. nan% nan% 0.0% nan% 0.9% nan% 0.2% 0.2% 0.3% 0.1% nan% 0.1%
Array A 25% 0.0 0.0 2.0 0.0 5.0 0.0 1.6 2.4 403.0 1.5 0.0 1.5 Array B Min 25% 0.0 0.0 0.0 0.0 1.0 2.0 0.0 0.0 4.0 5.0 0.0 0.0 1.0 1.6 0.3 1.4 108.0 265.0 2.2 2.8 0.0 0.0 2.2 2.8 Array C Min 25% 0.0 0.0 0.0 0.0 1.0 2.0 0.0 0.0 5.0 5.0 0.0 0.0 1.0 1.6 0.4 1.8 49.0 144.0 0.2 0.3 0.0 0.0 0.2 0.3 Array D Min 25% 0.0 0.0 0.0 0.0 1.0 2.0 0.0 0.0 5.0 5.0 0.0 0.0 1.0 1.6 0.5 4.2 134.0 402.0 0.1 0.2 0.0 0.0 0.1 0.2 Min 0.0 0.0 1.0 0.0 4.0 0.0 1.0 0.5 156.0 0.4 0.0 0.4
50% 0.0 0.0 8.0 0.0 5.0 0.0 3.9 7.2 1149.0 6.5 0.0 6.5
75% 0.0 0.0 16.0 0.0 3822.0 0.0 14.1 23.6 2043.0 9.2 0.0 9.2
90% 0.0 0.0 32.0 0.0 4815.0 0.0 15.2 35.7 2549.0 13.8 0.0 13.8
Max 0.0 0.0 64.0 0.0 5471.0 0.0 16.0 72.8 4253.0 18.6 0.0 18.6
Mean 0.0 0.0 15.8 0.0 1855.0 0.0 6.8 16.2 1473.0 6.7 0.0 6.7
50% 0.0 0.0 8.0 0.0 5.0 0.0 3.8 5.0 672.0 5.5 0.0 5.5
75% 0.0 0.0 16.0 0.0 3794.0 0.0 14.1 13.6 1528.0 10.6 0.0 10.6
90% 0.0 0.0 32.0 0.0 4832.0 0.0 14.9 39.3 2159.0 14.5 0.0 14.5
Max 0.0 0.0 64.0 0.0 5542.0 0.0 15.9 104.7 2929.0 20.7 0.0 20.7
Mean 0.0 0.0 15.8 0.0 1861.5 0.0 6.7 15.7 1020.9 7.9 0.0 7.9
50% 0.0 0.0 8.0 0.0 5.0 0.0 3.6 8.6 1743.0 1.9 0.0 1.9
75% 0.0 0.0 16.0 0.0 3820.0 0.0 6.9 49.9 6772.0 13.4 0.0 13.4
90% 0.0 0.0 32.0 0.0 4863.0 0.0 14.1 106.8 13918.0 25.9 0.0 25.9
Max 0.0 0.0 64.0 0.0 5520.0 0.0 14.8 109.6 25519.0 97.9 0.0 97.9
Mean 0.0 0.0 15.8 0.0 1856.3 0.0 5.7 35.2 5534.8 14.4 0.0 14.4
50% 0.0 0.0 8.0 0.0 5.0 0.0 3.5 21.3 1771.0 2.0 0.0 2.0
75% 0.0 0.0 16.0 0.0 3812.0 0.0 6.7 60.7 7760.0 7.5 0.0 7.5
90% 0.0 0.0 32.0 0.0 4795.0 0.0 14.1 107.0 15547.0 9.4 0.0 9.4
Max 0.0 0.0 64.1 0.0 5470.0 0.0 14.7 110.9 27676.0 12.8 0.0 12.8
Mean 0.0 0.0 15.8 0.0 1850.0 0.0 5.6 40.1 6542.6 4.5 0.0 4.5
Table F.3: Workload characteristics and performance are measured for each sample, on each storage device. The minimum value, percentiles, maximum value and average are reported for each measurement. In addition, the relative difference between the average performance of the best and second-to-best iteration is reported. This value quantifies the change in a given measurement across multiple runs of the same sample on the same storage device.
83
APPENDIX F. FITNESSCACHE MODEL TRAINING 1
120
Array A Array B Array C Array D
0.9 100 Bandwidth (MB/sec)
0.8 0.6 0.5 0.4 0.3
Array A Array B Array C Array D
0.1
80
60
40
20
0
0
80 60 40 20 0
120
0.2
100
Fraction
0.7
0
5
10
Bandwidth (MB/sec) 1
30000 25000 Throughput (IO/sec)
Fraction
0.7 0.6 0.5 0.4 Array A Array B Array C Array D 15000
10000
5000
0
0.1
0
5
10
80 Latency (ms)
0.6 0.4
35
70 60 50 40 30 20 10
60
0 50
30
20
10
0
40
Array A Array B Array C Array D
15 20 25 Sample number
Array A Array B Array C Array D
90 0.8
Fraction
30
0
100
Latency (ms)
35
10000 5000
1
0
30
15000
Throughput (IO/sec)
0.2
35
20000
25000
0.2
20000
0.3
30
Array A Array B Array C Array D
0.9 0.8
15 20 25 Sample number
0
5
10
15 20 25 Sample number
Figure F.1: The cumulative distribution of performance is shown (top). In addition, the performance of each array is shown (bottom), sorted by the performance of Array C.
84
Throughput (IO/sec) on Array A 0.25
0.3
0.2
0.25
0.4 0.3
0.15 0.1
0.2
0.05
0.1
0
<8
[8,
[15
15
0.35
Fraction
0.5
0
[ [ [ [ [ [ > 68 568, 979, 1391 1802 2214 2626 =30 3 97 1 , , , , 9) 391 180 221 262 303 7 ) 2) 4) 6) 7)
[23 [30 [37 [45 >= 5 , , , , 3) 30) 37) 45) 52) 2
Bandwidth (MB/sec) on Array B
0.3 0.2 0.1 0
<9
[9,
[18
18
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.1 0.05 0
<1
4
[14
[28 [41 [55 [68 [82 >= 9 , , , , , 8) 41) 55) 68) 82) 95) 5
,2
[10
[12 >= 1 , 2) 13) 3
,1
0.2
0.05 0
<4
[4,
6)
[6,
8)
[8,
10
)
[10
[12 [14 >= 1 , , 2) 14) 16) 6
,1
Latency (ms) on Array C
Fraction
<2
6
[27
[54
[81
[10
[13 [16 >= 1 46 4 4 8 5 2 , 5 4, 8 2, 1 39, 36, 34, 8932 44 1 0 1 1 1 4) 42) 839 353 623 893 ) 6) 4) 2)
0.4 0.3 0.2 0.1 0
<7
[7,
15
<2
[ [ [ [ [ [ > 97 2976 5818 8660 1150 1434 1718 =20 6 0 ,5 , , 2 3 5 81 866 115 , 14 , 17 , 20 27 8) 0) 02 34 18 0 ) 3) 5) 27)
)
[15
[22 [29 [37 [44 >= 5 , , , , 2) 29) 37) 44) 51) 1
,2
Latency (ms) on Array D
Fraction
0.2 0.15
)
0.15
Throughput (IO/sec) on Array D
Fraction
0.3 0.25
10
0.5
Bandwidth (MB/sec) on Array D 0.35
[8,
8)
0.6
[2 [4 [5 [6 [8 > , 2 7, 4 1, 5 4, 6 8, 8 1, 9 =95 7) 1) 4) 8) 1) 5)
0.55 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
[6,
0.1
[ [ [ [ [ [ > 37 437, 766, 1095 1424 1753 2082 =24 1 76 1 , , , , 6) 095 142 175 208 241 1 ) 4) 3) 2) 1)
[14
0.4
6)
0.25
Throughput (IO/sec) on Array C
Fraction
Fraction
4
[4,
Latency (ms) on Array B
<4
[27 [36 [45 [54 >= 6 , , , , 7) 36) 45) 54) 63) 3
0.55 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
4)
0.3
74
<1
[2,
0.35
Bandwidth (MB/sec) on Array C 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<2
0.4
,2
)
0
Fraction
Fraction
Fraction
0.4
0.05
Throughput (IO/sec) on Array B
0.6 0.5
0.2 0.15 0.1
<5
,2
)
0.7
Fraction
Latency (ms) on Array A
0.3
Fraction
Fraction
Bandwidth (MB/sec) on Array A 0.6
0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<2
Figure F.2: Probability distributions of performance.
[2,
3)
[3,
5)
[5,
6)
[6,
7)
[7,
9)
[9,
> 10 =10 )
85
APPENDIX F. FITNESSCACHE MODEL TRAINING
Array A to Array B (min 0.15, max 1.54) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .31 0.31 0.48 0.64 0.81 0.97 1.14 =1.3 ,0 , , , , , .48 0.64 0.81 0.97 1.14 1.30 0 ) ) ) ) ) )
Array A to Array C (min 0.17, max 12.66) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array B to Array A (min 0.65, max 6.86) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.5 0.4 0.3 0.2 0.1 0
<1
[ [ [ [ [ [ > .51 1.51 2.85 4.18 5.52 6.86 8.20 =9.5 ,2 , , , , , .85 4.18 5.52 6.86 8.20 9.53 3 ) ) ) ) ) )
<1
[ [ [ [ [ [ > .33 1.33 2.00 2.68 3.35 4.03 4.70 =5.3 ,2 , , , , , .00 2.68 3.35 4.03 4.70 5.38 8 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .55 0.55 1.01 1.48 1.94 2.41 2.88 =3.3 ,1 , , , , , .01 1.48 1.94 2.41 2.88 3.34 4 ) ) ) ) ) )
0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .20 0.20 0.33 0.46 0.59 0.72 0.85 =0.9 ,0 , , , , , .33 0.46 0.59 0.72 0.85 0.98 8 ) ) ) ) ) )
[ [ [ [ [ [ > .19 2.19 3.53 4.86 6.20 7.54 8.88 =10 ,3 , , , , , . .53 4.86 6.20 7.54 8.88 10.2 22 ) ) ) ) ) 2)
Array B to Array D (min 1.06, max 23.19) 0.6 0.5 0.4 0.3 0.2 0.1 0
<1
[ [ [ [ [ [ > .81 1.81 3.43 5.04 6.65 8.26 9.88 =11 ,3 , , , , , . .43 5.04 6.65 8.26 9.88 11.4 49 ) ) ) ) ) 9)
<2
[ [ [ [ [ [ > .90 2.90 4.73 6.57 8.40 10.2 12.0 =13 ,4 , , , 4 7 . .73 6.57 8.40 10.2 , 12 , 13 91 .07 .91 ) ) ) 4) ) )
Array C to Array B (min 0.07, max 4.95) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .48 0.48 0.89 1.30 1.71 2.12 2.52 =2.9 ,0 , , , , , .89 1.30 1.71 2.12 2.52 2.93 3 ) ) ) ) ) )
Array D to Array A (min 0.07, max 1.18) 0.25
<2
Array B to Array C (min 0.20, max 14.06) 0.55 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array A (min 0.08, max 5.83) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array A to Array D (min 0.85, max 13.60) 0.6
Array C to Array D (min 0.98, max 7.94) 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
<1
[ [ [ [ [ [ > .49 1.49 2.01 2.52 3.04 3.56 4.07 =4.5 ,2 , , , , , .01 2.52 3.04 3.56 4.07 4.59 9 ) ) ) ) ) )
Array D to Array B (min 0.04, max 0.94) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .14 0.14 0.23 0.33 0.43 0.52 0.62 =0.7 ,0 , , , , , .23 0.33 0.43 0.52 0.62 0.71 1 ) ) ) ) ) )
Array D to Array C (min 0.13, max 1.02) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .23 0.23 0.34 0.45 0.56 0.67 0.78 =0.8 ,0 , , , , , .34 0.45 0.56 0.67 0.78 0.89 9 ) ) ) ) ) )
Figure F.3: Each graph shows the probability distribution of the Bandwidth relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
86
Array A to Array B (min 0.15, max 1.48) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .31 0.31 0.47 0.63 0.79 0.95 1.11 =1.2 ,0 , , , , , .47 0.63 0.79 0.95 1.11 1.27 7 ) ) ) ) ) )
Array A to Array C (min 0.17, max 12.49) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array B to Array A (min 0.68, max 6.72) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.5 0.4 0.3 0.2 0.1 0
<1
[ [ [ [ [ [ > .52 1.52 2.86 4.21 5.56 6.91 8.25 =9.6 ,2 , , , , , .86 4.21 5.56 6.91 8.25 9.60 0 ) ) ) ) ) )
<1
[ [ [ [ [ [ > .35 1.35 2.01 2.68 3.35 4.02 4.69 =5.3 ,2 , , , , , .01 2.68 3.35 4.02 4.69 5.36 6 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .55 0.55 1.02 1.49 1.96 2.43 2.90 =3.3 ,1 , , , , , .02 1.49 1.96 2.43 2.90 3.37 7 ) ) ) ) ) )
<1
[ [ [ [ [ [ > .87 1.87 3.54 5.21 6.88 8.55 10.2 =11 ,3 , , , , 2 . .54 5.21 6.88 8.55 10.2 , 11 89 .89 ) ) ) ) 2) )
0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .20 0.20 0.34 0.47 0.60 0.73 0.86 =1.0 ,0 , , , , , .34 0.47 0.60 0.73 0.86 1.00 0 ) ) ) ) ) )
[ [ [ [ [ [ > .19 2.19 3.53 4.87 6.21 7.55 8.89 =10 ,3 , , , , , . .53 4.87 6.21 7.55 8.89 10.2 23 ) ) ) ) ) 3)
Array B to Array D (min 1.09, max 23.24) 0.6 0.5 0.4 0.3 0.2 0.1 0
<2
[ [ [ [ [ [ > .93 2.93 4.76 6.60 8.44 10.2 12.1 =13 ,4 , , , 7 1 . .76 6.60 8.44 10.2 , 12 , 13 95 .11 .95 ) ) ) 7) ) )
Array C to Array B (min 0.07, max 4.96) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .46 0.46 0.85 1.25 1.64 2.04 2.43 =2.8 ,0 , , , , , .85 1.25 1.64 2.04 2.43 2.82 2 ) ) ) ) ) )
Array D to Array A (min 0.07, max 1.17) 0.25
<2
Array B to Array C (min 0.20, max 14.83) 0.55 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array A (min 0.08, max 5.86) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array A to Array D (min 0.85, max 13.67) 0.6
Array C to Array D (min 0.97, max 7.94) 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
<1
[ [ [ [ [ [ > .49 1.49 2.00 2.52 3.04 3.55 4.07 =4.5 ,2 , , , , , .00 2.52 3.04 3.55 4.07 4.58 8 ) ) ) ) ) )
Array D to Array B (min 0.04, max 0.92) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .14 0.14 0.23 0.32 0.42 0.51 0.60 =0.7 ,0 , , , , , .23 0.32 0.42 0.51 0.60 0.70 0 ) ) ) ) ) )
Array D to Array C (min 0.13, max 1.03) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .23 0.23 0.34 0.45 0.56 0.67 0.77 =0.8 ,0 , , , , , .34 0.45 0.56 0.67 0.77 0.88 8 ) ) ) ) ) )
Figure F.4: Each graph shows the probability distribution of the Throughput relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
87
APPENDIX F. FITNESSCACHE MODEL TRAINING
Array A to Array B (min 0.66, max 6.57) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .34 1.34 2.01 2.69 3.37 4.04 4.72 =5.3 ,2 , , , , , .01 2.69 3.37 4.04 4.72 5.39 9 ) ) ) ) ) )
Array A to Array C (min 0.03, max 5.89) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .51 0.51 0.98 1.45 1.92 2.39 2.87 =3.3 ,0 , , , , , .98 1.45 1.92 2.39 2.87 3.34 4 ) ) ) ) ) )
Array B to Array A (min 0.15, max 1.51) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .31 0.31 0.48 0.64 0.80 0.96 1.13 =1.2 ,0 , , , , , .48 0.64 0.80 0.96 1.13 1.29 9 ) ) ) ) ) )
<3
[ [ [ [ [ [ > .20 3.20 6.24 9.27 12.3 15.3 18.3 =21 ,6 , , 0 4 7 . .24 9.27 12.3 , 15 , 18 , 21 40 .34 .37 .40 ) ) 0) ) ) )
<0
[ [ [ [ [ [ > .45 0.45 0.86 1.26 1.67 2.07 2.48 =2.8 ,0 , , , , , .86 1.26 1.67 2.07 2.48 2.88 8 ) ) ) ) ) )
<4
.99
[4.
[ [ [ [ [ > 99 9.13 13.2 17.4 21.5 25.7 =29 ,9 , 8 2 7 1 . .13 13.2 , 17 , 21 , 25 , 29 86 .42 .57 .71 .86 ) 8) ) ) ) )
[ [ [ [ [ [ > .16 0.16 0.30 0.44 0.57 0.71 0.85 =0.9 ,0 , , , , , .30 0.44 0.57 0.71 0.85 0.98 8 ) ) ) ) ) )
Array B to Array D (min 0.04, max 0.87) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .13 0.13 0.22 0.32 0.41 0.51 0.60 =0.7 ,0 , , , , , .22 0.32 0.41 0.51 0.60 0.70 0 ) ) ) ) ) )
Array C to Array B (min 0.20, max 20.56) 0.6 0.5 0.4 0.3 0.2 0.1 0
Array C to Array D (min 0.12, max 1.04) 0.35 0.3 0.25 0.2 0.15 0.1
<2
[ [ [ [ [ [ > .34 2.34 4.49 6.63 8.77 10.9 13.0 =15 ,4 , , , 1 6 . .49 6.63 8.77 10.9 , 13 , 15 20 .06 .20 ) ) ) 1) ) )
Array D to Array A (min 0.85, max 40.28) 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
<0
Array B to Array C (min 0.05, max 4.99) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array A (min 0.17, max 29.00) 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Array A to Array D (min 0.02, max 1.18) 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
0.05
<0
[ [ [ [ [ [ > .24 0.24 0.35 0.46 0.58 0.69 0.80 =0.9 ,0 , , , , , .35 0.46 0.58 0.69 0.80 0.92 2 ) ) ) ) ) )
Array D to Array B (min 1.15, max 28.56) 0.6 0.5 0.4 0.3 0.2 0.1 0
Array D to Array C (min 0.96, max 8.08) 0.6 0.5 0.4 0.3 0.2 0.1
<4
.27
[4.
[ [ [ [ [ > 27 7.39 10.5 13.6 16.7 19.8 =22 ,7 , 1 3 5 6 . .39 10.5 , 13 , 16 , 19 , 22 98 .63 .75 .86 .98 ) 1) ) ) ) )
0
<1
[ [ [ [ [ [ > .47 1.47 1.99 2.50 3.02 3.53 4.05 =4.5 ,1 , , , , , .99 2.50 3.02 3.53 4.05 4.56 6 ) ) ) ) ) )
Figure F.5: Each graph shows the probability distribution of the Latency relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
101
100.5
100.5
99.5
Array A Array B Array C Array D
Write size (KB) 101
100.5
100.5
99.5
Write queue
Array A Array B Array C Array D 0
-0.5
99 1
0
-0.5
99
0.5
Array A Array B Array C Array D
-1
99.5
100
Write jump (MB)
Figure F.6: The cumulative distribution of workload characteristics.
1
100
0.5
Fraction
101
-1
Fraction
Write fraction
0
-1
99 1
0
-0.5
-1
99
Array A Array B Array C Array D -0.5
99.5
100
1
100
0.5
Fraction
101
0.5
Fraction
88
89
APPENDIX F. FITNESSCACHE MODEL TRAINING
Overall Predictor Score Read jump 1.00 Read queue 0.91 Read size 0.81 Overall Predictor Score Read jump 1.00 Read queue 0.87 Read size 0.70 Overall Predictor Score Bandwidth 1.00 Latency 0.72 Throughput 0.46 Read queue 0.43 Read jump 0.31 Read size 0.25 Overall Predictor Score Latency 1.00 Throughput 0.52 Read jump 0.49 Bandwidth 0.41 Read queue 0.39 Read size 0.20
Absolute Bandwidth Throughput Predictor Score Predictor Score Read size 1.00 Read jump 1.00 Read jump 0.68 Read queue 0.83 Read queue 0.35 Read size 0.44 Relative Bandwidth Throughput Predictor Score Predictor Score Read size 1.00 Read jump 1.00 Read jump 0.71 Read queue 0.61 Read queue 0.39 Read size 0.34 Relative Performance Bandwidth Throughput Predictor Score Predictor Score Bandwidth 1.00 Throughput 1.00 Read jump 0.15 Read queue 0.52 Read queue 0.08 Read jump 0.51 Read size 0.08 Latency 0.33 Latency 0.07 Read size 0.24 Throughput 0.02 Bandwidth 0.23 Relative Fitness Bandwidth Throughput Predictor Score Predictor Score Latency 1.00 Latency 1.00 Read jump 0.48 Throughput 0.50 Throughput 0.42 Bandwidth 0.44 Bandwidth 0.41 Read jump 0.41 Read queue 0.30 Read queue 0.31 Read size 0.19 Read size 0.16
Latency Predictor Score Read queue 1.00 Read jump 0.80 Read size 0.57 Latency Predictor Score Read queue 1.00 Read jump 0.72 Read size 0.45 Latency Predictor Score Latency 1.00 Read queue 0.35 Throughput 0.22 Read size 0.19 Bandwidth 0.15 Read jump 0.04 Latency Predictor Score Latency 1.00 Throughput 0.69 Read jump 0.62 Read queue 0.61 Bandwidth 0.35 Read size 0.28
Table F.4: The normalized importance measure of each predictor.
Model Absolute Relative Relative Performance Relative Fitness
Overall 5.5 5.2 5.7 5.1
Absolute Array Array Array Array Relative Array Array Array Array Relative Performance Array Array Array Array Relative Fitness Array Array Array Array
Absolute Array A Array B Array C Array D
A 5.0 -
Relative Array A Array B Array C Array D
A 5.0 6.0 5.0 6.0
Relative Performance Array A Array B Array C Array D
A 6.0 4.0 7.0 5.0
Relative Fitness Array A Array B Array C Array D
A 1 1.0 4.0 6.0
Bandwidth 4.2 4.0 6.1 4.7
Pairwise Array 5.0 Array 5.0 5.3 5.3 5.7 Array 5.7 5.3 5.7 5.0 Array 1 4.3 5.0 5.7
Bandwidth B C 5.0 3.0 Bandwidth B C 1.0 3.0 5.0 3.0 1.0 3.0 6.0 3.0 Bandwidth B C 8.0 3.0 7.0 4.0 7.0 6.0 9.0 7.0 Bandwidth B C 5.0 5.0 1 4.0 5.0 1 6.0 5.0
D 4.0 D 5.0 5.0 4.0 4.0 D 7.0 6.0 6.0 6.0 D 3.0 7.0 5.0 1
Array 5.7 Array 4.3 5.7 3.7 4.7 Array 6.3 6.3 6.3 6.3 Array 6.0 1 5.3 4.7
Throughput 5.0 4.8 5.1 5.7
Array 6.0 Array 5.7 5.3 6.0 6.3 Array 3.0 6.3 7.3 6.7 Array 5.7 4.3 1 5.0
Throughput B C D 5.0 5.0 5.0 Throughput A B C D 5.0 5.0 6.0 5.0 5.0 5.0 4.0 5.0 5.0 5.0 5.0 5.0 5.0 3.0 5.0 5.0 Throughput A B C D 5.0 6.0 2.0 5.0 5.0 5.0 8.0 3.0 4.0 7.0 8.0 5.0 4.0 5.0 7.0 5.0 Throughput A B C D 1 6.0 6.0 8.0 6.0 1 7.0 4.0 5.0 5.0 1 5.0 5.0 6.0 5.0 1 A 5.0 -
Latency 7.2 6.9 6.0 4.9
Array 5.3 Array 5.7 5.7 5.3 5.3 Array 6.3 5.3 6.0 6.0 Array 5.7 5.0 4.3 1
A 5.0 A 5.0 5.0 6.0 6.0 A 6.0 7.0 6.0 6.0 A 1 6.0 6.0 6.0
Table F.5: Tree sizes (leaf nodes) and their averages.
Latency B C 7.0 10.0 Latency B C 7.0 8.0 7.0 9.0 5.0 10.0 5.0 11.0 Latency B C 5.0 4.0 7.0 7.0 5.0 8.0 5.0 6.0 Latency B C 7.0 6.0 1 2.0 6.0 1 2.0 5.0
D 7.0 D 7.0 7.0 7.0 7.0 D 7.0 7.0 7.0 7.0 D 6.0 4.0 3.0 1
91
APPENDIX F. FITNESSCACHE MODEL TRAINING
Bandwidth error (FitnessCache) Median relative error (Avg.)
Median relative error (Avg.)
Overall error (FitnessCache) 0.6 0.55 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
70
30 AM (i=j) AM
RF
0.6 0.55 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
RM RM’
70 RF
Latency error (FitnessCache) Median relative error (Avg.)
Median relative error (Avg.)
Throughput error (FitnessCache)
40 50 60 Samples used for training (%)
70 RF
0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
Figure F.7: Error vs. training. The training set size varies from 25% of the collected samples to 75%; the remaining samples are used for testing. For each training set size, the Median relative error is calculated for each pairing of devices. They are then averaged to produce the values shown.
92
Bandwidth error (FitnessCache) Mean relative error (Avg.)
Mean relative error (Avg.)
Overall error (FitnessCache) 3 2.5 2 1.5 1 0.5 0 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
7 6 5 4 3 2 1 0
70
30
RF
AM (i=j) AM
30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
RM RM’
70 RF
Latency error (FitnessCache)
1.2 1.1 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1
Mean relative error (Avg.)
Mean relative error (Avg.)
Throughput error (FitnessCache)
40 50 60 Samples used for training (%)
70 RF
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
Figure F.8: Error vs. training. The training set size varies from 25% of the collected samples to 75%; the remaining samples are used for testing. For each training set size, the Mean relative error is calculated for each pairing of devices. They are then averaged to produce the values shown.
Appendix G
Postmark model training
Application Postmark (phase 1) (pmc) Postmark (phase 2) (pmt) Total used
Samples 50 50
Iters 3 3
First sample 0 0 50
Last sample 24 24
Table G.1: Multiple training samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Array A 0.76 110.72 53.72 1754.38 3654.38 44.46 2.23 27.39 278.92 50.74 75.90 8.76
Array B 0.77 109.48 54.12 1722.18 3631.52 28.20 1.95 22.20 232.44 28.62 38.05 27.90
Array C 0.76 108.27 51.53 1518.64 3784.66 32.88 1.66 14.48 165.42 110.14 175.20 7.42
Array D 0.77 109.50 54.11 1771.54 3558.88 27.34 1.54 38.62 421.64 19.82 23.71 10.88
MAD 0.0 0.6 0.9 86.5 63.6 5.6 0.2 7.3 75.7 28.9 48.5 7.1
COV 0.7% 0.8% 2.0% 6.0% 2.2% 20.5% 14.5% 34.2% 34.2% 67.3% 75.6% 60.2%
Max. Diff. 1.3% 2.3% 5.0% 16.6% 6.3% 62.6% 44.8% 166.7% 154.9% 455.7% 638.9% 276.0%
Table G.2: Workload characteristics and performance are measured for each sample, on each storage device. The average value for each measurement is reported in this table. The mean absolute deviation (MAD), coefficient of variation (COV), and maximum relative differences of these averages are also reported; these metrics quantify how the averages change among the storage devices. For example, the mean average deviation of the averages for Read latency (ms) is 7.08, their coeefficient of variation is 60.169%, and the maximum relative difference is 276.01%.
93
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 1.0% 2.8% 1.2% 3.6% 2.1% 2.3% 23.1% 3.0% 0.4% 1.8% 3.5% 14.7%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 0.3% 3.2% 2.0% 3.4% 0.5% 3.0% 0.9% 3.4% 0.3% 1.1% 1.2% 14.4%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 0.1% 5.8% 5.0% 3.4% 1.0% 3.6% 5.2% 2.3% 3.5% 3.2% 2.5% 3.0%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 0.3% 1.9% 0.3% 4.2% 0.2% 2.9% 2.2% 2.1% 1.2% 0.9% 1.8% 0.0%
Array A Min 25% 0.5 0.6 29.4 88.8 4.0 4.0 391.0 1566.0 921.0 1373.0 17.1 25.9 1.0 1.1 7.0 22.2 214.0 235.0 30.9 34.2 50.0 59.5 4.8 5.4 Array B Min 25% 0.5 0.6 25.6 90.0 4.0 4.0 388.0 1545.0 1071.0 1423.0 16.7 21.0 1.0 1.1 5.7 17.2 136.0 155.0 12.7 19.5 22.2 25.7 7.9 13.4 Array C Min 25% 0.5 0.6 23.1 88.4 4.0 4.0 307.0 1322.0 1139.0 1666.0 16.4 24.3 1.0 1.0 4.6 13.8 124.0 145.0 46.2 68.3 116.5 127.3 0.8 4.0 Array D Min 25% 0.6 0.6 42.0 87.9 4.0 4.0 549.0 1571.0 980.0 1306.0 16.5 19.6 1.0 1.1 6.5 19.9 167.0 193.0 9.1 14.7 15.6 19.6 3.1 8.3
50% 0.7 118.6 4.0 1688.0 2123.0 49.5 1.2 28.3 279.0 44.6 64.4 5.8
75% 0.9 130.9 103.8 1861.0 5815.0 53.1 1.2 31.9 307.0 55.4 88.9 6.0
90% 0.9 143.4 139.8 2124.0 5947.0 61.1 1.3 37.4 329.0 75.6 105.0 6.7
Max 1.0 148.0 152.4 3166.0 6304.0 84.2 31.7 40.3 372.0 109.2 156.7 89.7
Mean 0.8 110.7 53.7 1754.4 3654.4 44.5 2.2 27.4 278.9 50.7 75.9 8.8
50% 0.7 114.2 4.1 1657.0 2386.0 23.4 1.2 19.0 247.0 23.0 29.1 15.7
75% 0.9 127.6 108.5 1858.0 5903.0 29.9 1.2 29.8 283.0 25.8 31.4 30.6
90% 0.9 137.1 141.5 2121.0 5993.0 49.7 1.3 32.4 313.0 34.5 55.7 37.5
Max 1.0 152.1 155.1 3161.0 6115.0 57.6 20.1 36.6 405.0 130.1 175.0 228.5
Mean 0.8 109.5 54.1 1722.2 3631.5 28.2 1.9 22.2 232.4 28.6 38.1 27.9
50% 0.7 112.1 4.0 1591.0 2101.0 31.1 1.3 15.4 158.0 91.6 141.3 7.4
75% 0.9 129.0 103.0 1707.0 6050.0 42.6 1.3 16.2 174.0 136.6 205.1 9.0
90% 0.9 139.6 131.8 1970.0 6182.0 46.4 1.5 17.3 205.0 197.2 288.1 10.4
Max 1.0 151.1 152.5 2317.0 6406.0 53.6 11.3 18.5 243.0 262.8 384.9 27.5
Mean 0.8 108.3 51.5 1518.6 3784.7 32.9 1.7 14.5 165.4 110.1 175.2 7.4
50% 0.7 116.8 4.0 1712.0 2033.0 22.8 1.2 25.0 329.0 17.3 21.4 11.5
75% 0.9 128.6 108.4 1850.0 5791.0 29.2 1.2 54.1 549.0 21.8 23.6 13.0
90% 0.9 143.7 141.9 2298.0 5844.0 50.4 1.3 66.6 804.0 27.9 30.2 13.9
Max 1.0 150.1 152.6 3028.0 6310.0 56.2 9.1 74.4 1285.0 43.5 54.1 17.6
Mean 0.8 109.5 54.1 1771.5 3558.9 27.3 1.5 38.6 421.6 19.8 23.7 10.9
Table G.3: Workload characteristics and performance are measured for each sample, on each storage device. The minimum value, percentiles, maximum value and average are reported for each measurement. In addition, the relative difference between the average performance of the best and second-to-best iteration is reported. This value quantifies the change in a given measurement across multiple runs of the same sample on the same storage device.
95
APPENDIX G. POSTMARK MODEL TRAINING 1
80
Array A Array B Array C Array D
70 Bandwidth (MB/sec)
0.6 0.4 Array A Array B Array C Array D
50 40 30 20
0
50
40
30
20
0
10
0
60
10 0
70
0.2
60
Fraction
0.8
5
10
Bandwidth (MB/sec) 1
1400
Throughput (IO/sec)
0.4 Array A Array B Array C Array D
1000
400
1000
0 0
250 Latency (ms)
0.8 0.6 0.4
5
10 15 20 25 30 35 40 45 50 Sample number
Array A Array B Array C Array D
200 150 100
220
200
50 180
120
100
80
60
40
0
20
160
Array A Array B Array C Array D 140
Fraction
300
Latency (ms)
50
600
1
0
45
800
Throughput (IO/sec)
0.2
40
200 900
600
500
400
300
200
0
100
0
800
0.2
700
Fraction
0.6
20 25 30 35 Sample number
Array A Array B Array C Array D
1200
0.8
15
0 0
5
10
15
20 25 30 35 Sample number
40
45
50
Figure G.1: The cumulative distribution of performance is shown (top). In addition, the performance of each array is shown (bottom), sorted by the performance of Array C.
96
Throughput (IO/sec) on Array A 0.25
0.2
0.2
0.15 0.1
Latency (ms) on Array A 0.4 0.35 0.3
0.15
Fraction
Fraction
Fraction
Bandwidth (MB/sec) on Array A 0.25
0.1 0.05 0
<1
1
[11
[15 [19 [22 [26 [30 >= 3 , , , , , 5) 19) 22) 26) 30) 34) 4
[ [ [ [ [ [ > 32 232, 250, 268, 286, 305, 323, =34 1 25 2 2 3 3 3 0) 68) 86) 05) 23) 41)
,1
0.3
0.25
0.35
0.2 Fraction
0.3
0.15 0.1
0
<9
[9,
13
)
[13
<1
63
[16 [20 [23 [27 >= 3 , , , , 6) 20) 23) 27) 30) 0
,1
Fraction
0.3 0.2 0.1 0.05 0
<6
[6,
[8,
8)
9)
[9,
11
)
[11
0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
[13 [14 >= 1 , , 3) 14) 16) 6
0
4
[14
[22 [29 [37 [44 [52 >= 5 , , , , , 2) 29) 37) 44) 52) 59) 9
,2
[22 [26 [31 [35 [39 >= 4 , , , , , 2) 26) 31) 35) 39) 44) 4
,2
Latency (ms) on Array C 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<6
6
0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
[66
[85 [10 [12 [14 [16 >= 1 , 4 4 3 3 5) 104 , 12 , 14 , 16 , 18 82 ) 4) 3) 3) 2)
,8
Latency (ms) on Array D 0.3 0.25 Fraction
Fraction
<1
[17
Throughput (IO/sec) on Array D
0.2
0.1
<1
7
[ [ [ [ [ [ > 36 136, 148, 159, 171, 183, 194, =20 6 14 1 1 1 1 2 8) 59) 71) 83) 94) 06)
Bandwidth (MB/sec) on Array D 0.25
0.05
0
<1
,1
0.15
[1 [2 [2 [2 [2 > 3, 90, 16, 43, 70, 96, =323 19 2 2 2 2 3 0) 16) 43) 70) 96) 23)
Throughput (IO/sec) on Array C
0.35
0.15
0.2 0.15 0.05
[16
Fraction
Bandwidth (MB/sec) on Array C 0.4
0.25
0.25
0.1
0.05
0.05 0
[48 [56 [64 [73 [81 >= 9 , , , , , 8) 56) 64) 73) 81) 90) 0
,4
Latency (ms) on Array B
0.25
0.15
[39
Throughput (IO/sec) on Array B 0.4
0.2
<3
9
0.3
Fraction
Fraction
0
0.35
0.1
Fraction
0.05
<2
Bandwidth (MB/sec) on Array B
Fraction
0.2 0.15 0.1
0.05 0
0.25
0.2 0.15 0.1 0.05
<2
[ [ [ [ [ [ > 63 263, 360, 456, 552, 649, 745, =84 2 36 4 5 6 7 8 0) 56) 52) 49) 45) 42)
0
<1
Figure G.2: Probability distributions of performance.
3
[13
[16 [20 [23 [27 [30 >= 3 , , , , , 6) 20) 23) 27) 30) 34) 4
,1
97
APPENDIX G. POSTMARK MODEL TRAINING
Array A to Array B (min 0.54, max 1.12) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
Array A to Array C (min 0.41, max 0.77) 0.3 0.25 0.2 0.15 0.1 0.05
<0
[ [ [ [ [ [ > .61 0.61 0.67 0.74 0.81 0.88 0.94 =1.0 ,0 , , , , , .67 0.74 0.81 0.88 0.94 1.01 1 ) ) ) ) ) )
0
<0
[ [ [ [ [ [ > .44 0.44 0.47 0.50 0.53 0.56 0.60 =0.6 ,0 , , , , , .47 0.50 0.53 0.56 0.60 0.63 3 ) ) ) ) ) )
Array B to Array A (min 0.89, max 1.86) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
0.25 0.2 0.15 0.1 0.05
[ [ [ [ [ [ > .99 0.99 1.09 1.19 1.30 1.40 1.50 =1.6 ,1 , , , , , .09 1.19 1.30 1.40 1.50 1.60 0 ) ) ) ) ) )
0
<0
[ [ [ [ [ [ > .53 0.53 0.59 0.65 0.71 0.78 0.84 =0.9 ,0 , , , , , .59 0.65 0.71 0.78 0.84 0.90 0 ) ) ) ) ) )
Array C to Array A (min 1.29, max 2.43) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.25 0.2 0.15 0.1 0.05
[ [ [ [ [ [ > .43 1.43 1.56 1.69 1.82 1.95 2.08 =2.2 ,1 , , , , , .56 1.69 1.82 1.95 2.08 2.21 1 ) ) ) ) ) )
0
<1
[ [ [ [ [ [ > .09 1.09 1.24 1.38 1.52 1.66 1.80 =1.9 ,1 , , , , , .24 1.38 1.52 1.66 1.80 1.94 4 ) ) ) ) ) )
Array D to Array A (min 0.32, max 1.37) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .45 0.45 0.57 0.69 0.82 0.94 1.06 =1.1 ,0 , , , , , .57 0.69 0.82 0.94 1.06 1.19 9 ) ) ) ) ) )
[ [ [ [ [ [ > .00 1.00 1.26 1.53 1.80 2.07 2.34 =2.6 ,1 , , , , , .26 1.53 1.80 2.07 2.34 2.61 1 ) ) ) ) ) )
Array B to Array D (min 1.07, max 2.87) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .29 1.29 1.50 1.71 1.92 2.13 2.34 =2.5 ,1 , , , , , .50 1.71 1.92 2.13 2.34 2.55 5 ) ) ) ) ) )
Array C to Array B (min 0.95, max 2.11) 0.3
<1
<1
Array B to Array C (min 0.47, max 1.05) 0.3
<0
Array A to Array D (min 0.73, max 3.10) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array D (min 1.29, max 4.71) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .69 1.69 2.08 2.48 2.87 3.27 3.67 =4.0 ,2 , , , , , .08 2.48 2.87 3.27 3.67 4.06 6 ) ) ) ) ) )
Array D to Array B (min 0.35, max 0.93) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<0
[ [ [ [ [ [ > .42 0.42 0.48 0.55 0.61 0.68 0.75 =0.8 ,0 , , , , , .48 0.55 0.61 0.68 0.75 0.81 1 ) ) ) ) ) )
Array D to Array C (min 0.21, max 0.78) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .28 0.28 0.35 0.41 0.48 0.54 0.61 =0.6 ,0 , , , , , .35 0.41 0.48 0.54 0.61 0.68 8 ) ) ) ) ) )
Figure G.3: Each graph shows the probability distribution of the Bandwidth relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
98
Array A to Array B (min 0.57, max 1.11) 0.18 0.17 0.16 0.15 0.14 0.13 0.12 0.11 0.1 0.09 0.08
<0
[ [ [ [ [ [ > .63 0.63 0.69 0.74 0.80 0.85 0.91 =0.9 ,0 , , , , , .69 0.74 0.80 0.85 0.91 0.96 6 ) ) ) ) ) )
Array A to Array C (min 0.41, max 0.79) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<0
[ [ [ [ [ [ > .46 0.46 0.50 0.54 0.59 0.63 0.67 =0.7 ,0 , , , , , .50 0.54 0.59 0.63 0.67 0.72 2 ) ) ) ) ) )
Array B to Array A (min 0.90, max 1.74) 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<0
[ [ [ [ [ [ > .99 0.99 1.09 1.18 1.28 1.37 1.46 =1.5 ,1 , , , , , .09 1.18 1.28 1.37 1.46 1.56 6 ) ) ) ) ) )
<1
[ [ [ [ [ [ > .39 1.39 1.51 1.63 1.75 1.87 1.99 =2.1 ,1 , , , , , .51 1.63 1.75 1.87 1.99 2.12 2 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .52 0.52 0.61 0.70 0.79 0.88 0.96 =1.0 ,0 , , , , , .61 0.70 0.79 0.88 0.96 1.05 5 ) ) ) ) ) )
[ [ [ [ [ [ > .00 1.00 1.26 1.51 1.76 2.01 2.26 =2.5 ,1 , , , , , .26 1.51 1.76 2.01 2.26 2.52 2 ) ) ) ) ) )
Array B to Array D (min 1.06, max 3.78) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .28 1.28 1.50 1.72 1.94 2.16 2.38 =2.6 ,1 , , , , , .50 1.72 1.94 2.16 2.38 2.60 0 ) ) ) ) ) )
Array C to Array B (min 0.75, max 2.32) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<0
[ [ [ [ [ [ > .90 0.90 1.06 1.22 1.37 1.53 1.69 =1.8 ,1 , , , , , .06 1.22 1.37 1.53 1.69 1.84 4 ) ) ) ) ) )
Array D to Array A (min 0.28, max 1.33) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
Array B to Array C (min 0.43, max 1.34) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
Array C to Array A (min 1.27, max 2.43) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08
Array A to Array D (min 0.75, max 3.51) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array D (min 1.02, max 5.87) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .50 1.50 1.99 2.47 2.95 3.44 3.92 =4.4 ,1 , , , , , .99 2.47 2.95 3.44 3.92 4.40 0 ) ) ) ) ) )
Array D to Array B (min 0.26, max 0.94) 0.3
Array D to Array C (min 0.17, max 0.98) 0.25
0.25
0.2
0.2
0.15
0.15 0.1
0.1
0.05
0.05
<0
[ [ [ [ [ [ > .41 0.41 0.54 0.66 0.79 0.91 1.04 =1.1 ,0 , , , , , .54 0.66 0.79 0.91 1.04 1.17 7 ) ) ) ) ) )
0
<0
[ [ [ [ [ [ > .34 0.34 0.42 0.49 0.57 0.65 0.72 =0.8 ,0 , , , , , .42 0.49 0.57 0.65 0.72 0.80 0 ) ) ) ) ) )
0
<0
[ [ [ [ [ [ > .26 0.26 0.35 0.45 0.54 0.63 0.72 =0.8 ,0 , , , , , .35 0.45 0.54 0.63 0.72 0.81 1 ) ) ) ) ) )
Figure G.4: Each graph shows the probability distribution of the Throughput relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
99
APPENDIX G. POSTMARK MODEL TRAINING
Array A to Array B (min 0.32, max 1.22) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<0
[ [ [ [ [ [ > .37 0.37 0.43 0.48 0.53 0.58 0.64 =0.6 ,0 , , , , , .43 0.48 0.53 0.58 0.64 0.69 9 ) ) ) ) ) )
Array A to Array C (min 1.03, max 4.72) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array B to Array A (min 0.82, max 3.10) 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .08 1.08 1.35 1.62 1.88 2.15 2.41 =2.6 ,1 , , , , , .35 1.62 1.88 2.15 2.41 2.68 8 ) ) ) ) ) )
<1
[ [ [ [ [ [ > .23 1.23 1.44 1.65 1.85 2.06 2.27 =2.4 ,1 , , , , , .44 1.65 1.85 2.06 2.27 2.48 8 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .27 0.27 0.32 0.38 0.43 0.49 0.54 =0.6 ,0 , , , , , .32 0.38 0.43 0.49 0.54 0.60 0 ) ) ) ) ) )
<2
[ [ [ [ [ [ > .29 2.29 3.06 3.84 4.61 5.38 6.15 =6.9 ,3 , , , , , .06 3.84 4.61 5.38 6.15 6.93 3 ) ) ) ) ) )
0.25
0.25
0.2
0.2
0.15
0.15
0.1
0.1
0.05
0.05
0
<0
[ [ [ [ [ [ > .12 0.12 0.15 0.18 0.22 0.25 0.28 =0.3 ,0 , , , , , .15 0.18 0.22 0.25 0.28 0.31 1 ) ) ) ) ) )
0.14
0.2
0.12
0.15
0.1
0.1
0.08 0.06
.11
[ [ [ [ [ > 11 2.37 2.62 2.88 3.13 3.39 =3.6 ,2 , , , , , .37 2.62 2.88 3.13 3.39 3.64 4 ) ) ) ) ) )
0
Array D to Array B (min 1.09, max 3.34)
0.25
[2.
[ [ [ [ [ [ > .37 0.37 0.45 0.52 0.59 0.67 0.74 =0.8 ,0 , , , , , .45 0.52 0.59 0.67 0.74 0.81 1 ) ) ) ) ) )
Array C to Array D (min 0.08, max 0.26) 0.3
0.16
<2
<0
Array C to Array B (min 0.09, max 0.66)
Array D to Array A (min 1.86, max 4.25)
0
[ [ [ [ [ [ > .27 0.27 0.30 0.34 0.37 0.41 0.44 =0.4 ,0 , , , , , .30 0.34 0.37 0.41 0.44 0.48 8 ) ) ) ) ) )
Array B to Array D (min 0.30, max 0.92) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.3
0.3
0.05
<0
Array B to Array C (min 1.52, max 11.50) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array A (min 0.21, max 0.97) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array A to Array D (min 0.24, max 0.54) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08
0.04
<1
[ [ [ [ [ [ > .15 1.15 1.21 1.26 1.32 1.38 1.43 =1.4 ,1 , , , , , .21 1.26 1.32 1.38 1.43 1.49 9 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .10 0.10 0.12 0.14 0.16 0.18 0.20 =0.2 ,0 , , , , , .12 0.14 0.16 0.18 0.20 0.22 2 ) ) ) ) ) )
Array D to Array C (min 3.83, max 12.98) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<4
[ [ [ [ [ [ > .48 4.48 5.13 5.78 6.43 7.08 7.73 =8.3 ,5 , , , , , .13 5.78 6.43 7.08 7.73 8.38 8 ) ) ) ) ) )
Figure G.5: Each graph shows the probability distribution of the Latency relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
1 0.9
0.8
0.8
0.7
0.7
0.6
0.6
0.3
0.8
0.8
0.6
0.6
80
Write jump (MB)
Figure G.6: The cumulative distribution of workload characteristics.
3000
1500
1000
500
80
0
0 70
60
50
40
30
10
20
Write queue
Array A Array B Array C Array D
0.2
Array A Array B Array C Array D
0
0.4
2500
0.4
2000
Fraction
1
0.2
60
Write size (KB)
1
0
Fraction
Write fraction
40
0
0
160
0.1 1
0.6
0.5
0.4
0.3
0.2
0.1
0
0
Array A Array B Array C Array D
0.2
0.9
0.1
0.8
Array A Array B Array C Array D
0.2
0.4
20
0.3
0.5
140
0.4
120
0.5
100
Fraction
1 0.9
0.7
Fraction
100
101
APPENDIX G. POSTMARK MODEL TRAINING
Overall Predictor Write fraction Write size Read size Write queue Read jump
Score 1.00 0.92 0.49 0.44 0.25
Overall Predictor Write fraction Write size Read queue Read size Write queue Write jump Read jump
Score 1.00 0.77 0.41 0.41 0.28 0.24 0.18
Overall Predictor Write fraction Read size Write size Bandwidth Latency Read queue Throughput Write queue Write jump Write latency Read jump
Score 1.00 0.65 0.65 0.63 0.51 0.49 0.36 0.25 0.19 0.11 0.10
Overall Predictor Write fraction Write size Bandwidth Read queue Read size Write queue Throughput Latency Read jump Write jump Write latency Read latency
Score 1.00 0.58 0.52 0.50 0.44 0.35 0.32 0.26 0.25 0.21 0.17 0.10
Absolute Bandwidth Throughput Predictor Score Predictor Score Write size 1.00 Read size 1.00 Write fraction 0.76 Write size 0.62 Read size 0.27 Write jump 0.48 Read jump 0.22 Write queue 0.44 Write queue 0.15 Read queue 0.18 Relative Bandwidth Throughput Predictor Score Predictor Score Write fraction 1.00 Read size 1.00 Write size 0.99 Write fraction 0.91 Read size 0.27 Write size 0.83 Read queue 0.24 Write jump 0.53 Write queue 0.19 Read queue 0.49 Read jump 0.10 Write queue 0.48 Write jump 0.03 Read jump 0.14 Relative Performance Bandwidth Throughput Predictor Score Predictor Score Write fraction 1.00 Read size 1.00 Bandwidth 0.97 Write size 0.50 Write size 0.82 Write queue 0.33 Read queue 0.26 Read queue 0.33 Read size 0.23 Throughput 0.31 Latency 0.18 Latency 0.31 Write queue 0.11 Write fraction 0.28 Write latency 0.06 Write jump 0.26 Throughput 0.06 Bandwidth 0.08 Read latency 0.05 Read latency 0.07 Write jump 0.02 Read jump 0.06 Relative Fitness Bandwidth Throughput Predictor Score Predictor Score Write fraction 1.00 Write fraction 1.00 Bandwidth 0.47 Write size 0.38 Read size 0.47 Bandwidth 0.33 Throughput 0.43 Read queue 0.32 Write size 0.39 Read jump 0.32 Read queue 0.38 Write queue 0.26 Write queue 0.29 Throughput 0.22 Read jump 0.19 Latency 0.15 Write jump 0.13 Write jump 0.11 Write latency 0.11 Read size 0.09 Latency 0.11 Read latency 0.08 Read latency 0.04 Write latency 0.05
Latency Predictor Write fraction Write queue Write size Read queue Read jump
Score 1.00 0.40 0.30 0.25 0.25
Latency Predictor Write fraction Read queue Write size Write jump Read jump Write queue Read size
Score 1.00 0.54 0.44 0.31 0.29 0.25 0.20
Latency Predictor Write fraction Latency Read queue Throughput Read size Bandwidth Write jump Write latency Read jump Write queue Write size
Score 1.00 0.79 0.59 0.54 0.38 0.28 0.22 0.22 0.21 0.15 0.09
Latency Predictor Write size Read size Read queue Bandwidth Latency Write fraction Write queue Write jump Write latency Read latency Throughput Read jump
Score 1.00 0.83 0.82 0.73 0.60 0.58 0.47 0.47 0.41 0.21 0.19 0.12
Table G.4: The normalized importance measure of each predictor.
Model Absolute Relative Relative Performance Relative Fitness
Overall 7.7 8.0 6.8 6.2
Absolute Array Array Array Array Relative Array Array Array Array Relative Performance Array Array Array Array Relative Fitness Array Array Array Array
Absolute Array A Array B Array C Array D
A 10.0 -
Relative Array A Array B Array C Array D
A 10.0 10.0 4.0 12.0
Relative Performance Array A Array B Array C Array D
A 9.0 7.0 8.0 9.0
Relative Fitness Array A Array B Array C Array D
A 1 5.0 11.0 7.0
Bandwidth B C 11.0 10.0 Bandwidth B C 10.0 10.0 11.0 11.0 10.0 10.0 11.0 12.0 Bandwidth B C 10.0 8.0 9.0 9.0 8.0 7.0 12.0 9.0 Bandwidth B C 8.0 4.0 1 4.0 6.0 1 8.0 3.0
Bandwidth 9.8 9.3 8.4 6.5
Pairwise Array 6.0 Array 6.0 8.7 5.0 7.7 Array 8.3 5.3 5.3 6.0 Array 1 8.0 7.0 6.3
D 8.0 D 7.0 7.0 8.0 8.0 D 7.0 7.0 7.0 7.0 D 7.0 7.0 8.0 1
Array 8.0 Array 8.3 8.0 10.3 9.0 Array 6.3 8.3 9.0 9.7 Array 7.3 1 6.3 6.0
Throughput 6.2 6.8 5.4 6.2
Array 9.0 Array 6.3 10.7 9.0 5.7 Array 5.0 6.3 8.0 6.0 Array 4.3 3.3 1 4.7
Throughput B C 8.0 7.0 Throughput A B C 2.0 8.0 3.0 9.0 8.0 11.0 4.0 8.0 7.0 2.0 9.0 2.0 Throughput A B C 8.0 4.0 2.0 3.0 7.0 4.0 4.0 8.0 8.0 3.0 9.0 4.0 Throughput A B C 1 6.0 2.0 9.0 1 4.0 6.0 9.0 1 8.0 7.0 3.0 A 2.0 -
Latency 7.2 7.8 6.7 5.8
Array 8.0 Array 8.0 7.7 8.3 8.0 Array 7.7 7.3 8.0 8.3 Array 6.3 6.7 7.3 1
D 8.0
A 6.0 -
D 7.0 9.0 9.0 8.0
A 6.0 7.0 7.0 9.0
D 8.0 8.0 8.0 9.0
A 8.0 6.0 4.0 6.0
D 8.0 6.0 6.0 1
A 1 10.0 4.0 4.0
Table G.5: Tree sizes (leaf nodes) and their averages.
Latency B C 5.0 10.0 Latency B C 7.0 6.0 5.0 10.0 13.0 10.0 7.0 3.0 Latency B C 5.0 5.0 9.0 6.0 11.0 9.0 8.0 5.0 Latency B C 8.0 7.0 1 2.0 4.0 1 3.0 8.0
D 8.0 D 10.0 7.0 8.0 8.0 D 8.0 7.0 9.0 9.0 D 4.0 7.0 8.0 1
103
APPENDIX G. POSTMARK MODEL TRAINING
Bandwidth error (Postmark) Median relative error (Avg.)
Median relative error (Avg.)
Overall error (Postmark) 0.12 0.11 0.1 0.09 0.08 0.07 0.06 0.05 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
0.095 0.09 0.085 0.08 0.075 0.07 0.065 0.06 0.055 0.05 0.045 0.04
70
30 AM (i=j) AM
RF
0.1 0.095 0.09 0.085 0.08 0.075 0.07 0.065 0.06 0.055 0.05 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
RM RM’
70 RF
Latency error (Postmark) Median relative error (Avg.)
Median relative error (Avg.)
Throughput error (Postmark)
40 50 60 Samples used for training (%)
70 RF
0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
Figure G.7: Error vs. training. The training set size varies from 25% of the collected samples to 75%; the remaining samples are used for testing. For each training set size, the Median relative error is calculated for each pairing of devices. They are then averaged to produce the values shown.
104
Bandwidth error (Postmark) Mean relative error (Avg.)
Mean relative error (Avg.)
Overall error (Postmark) 0.28 0.26 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05
70
30
RF
AM (i=j) AM
30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
RM RM’
70 RF
Latency error (Postmark)
0.16 0.15 0.14 0.13 0.12 0.11 0.1 0.09 0.08 0.07
Mean relative error (Avg.)
Mean relative error (Avg.)
Throughput error (Postmark)
40 50 60 Samples used for training (%)
70 RF
0.35 0.3 0.25 0.2 0.15 0.1 0.05 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
Figure G.8: Error vs. training. The training set size varies from 25% of the collected samples to 75%; the remaining samples are used for testing. For each training set size, the Mean relative error is calculated for each pairing of devices. They are then averaged to produce the values shown.
Appendix H
Cello model training
Application (srt) Total used
Samples 78
Iters 3
First sample 0 39
Last sample 38
Table H.1: Multiple training samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Array A 0.45 17.70 6.98 8.64 313.28 57.95 5.99 8.60 807.82 19.10 41.15 6.98
Array B 0.44 21.41 7.14 8.79 303.72 51.73 5.84 4.33 435.87 44.86 123.67 9.35
Array C 0.44 17.67 7.37 9.15 301.67 53.74 5.76 5.04 481.05 42.51 108.74 9.14
Array D 0.47 20.69 7.22 12.44 326.10 59.93 6.08 10.03 974.33 7.40 9.10 10.14
MAD 0.0 1.7 0.1 1.3 8.5 3.1 0.1 2.3 216.3 15.2 45.5 1.0
COV 2.8% 8.8% 1.9% 16.0% 3.1% 5.8% 2.1% 34.1% 33.3% 55.5% 66.8% 13.2%
Max. Diff. 6.8% 21.2% 5.6% 44.0% 8.1% 15.8% 5.6% 131.6% 123.5% 506.2% 1259.0% 45.3%
Table H.2: Workload characteristics and performance are measured for each sample, on each storage device. The average value for each measurement is reported in this table. The mean absolute deviation (MAD), coefficient of variation (COV), and maximum relative differences of these averages are also reported; these metrics quantify how the averages change among the storage devices. For example, the mean average deviation of the averages for Read latency (ms) is 0.96, their coeefficient of variation is 13.160%, and the maximum relative difference is 45.27%.
105
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 10.7% 1.5% 0.1% 14.5% 0.5% 0.1% 0.1% 5.0% 2.8% 8.2% 0.1% 0.5%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 7.8% 6.7% 0.3% 0.4% 0.7% 1.4% 0.2% 7.6% 0.4% 8.5% 1.9% 0.3%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 6.2% 4.9% 0.5% 0.8% 1.1% 0.3% 0.1% 4.9% 0.1% 5.8% 1.0% 0.1%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 7.0% 3.4% 0.6% 21.7% 1.0% 1.5% 0.2% 18.0% 18.4% 6.9% 0.2% 16.9%
Array A Min 25% 0.2 0.4 11.3 13.7 5.7 6.1 0.0 5.0 106.0 200.0 52.2 56.3 1.0 1.6 4.4 6.2 460.0 588.0 7.6 17.3 34.0 38.1 2.1 3.5 Array B Min 25% 0.1 0.3 9.2 12.2 5.3 6.0 0.0 0.0 112.0 200.0 42.8 49.9 1.0 1.6 1.8 3.1 218.0 312.0 6.7 30.0 106.9 117.7 4.7 6.1 Array C Min 25% 0.1 0.4 9.5 12.2 5.4 6.1 0.0 0.0 113.0 209.0 48.6 51.8 1.0 1.6 2.4 4.0 282.0 374.0 17.9 31.6 99.0 103.4 2.5 4.4 Array D Min 25% 0.2 0.4 11.7 13.7 5.8 6.2 4.0 5.0 110.0 224.0 52.0 58.8 1.0 1.7 4.3 7.2 401.0 744.0 4.0 5.2 6.9 7.5 3.7 6.0
50% 0.5 16.2 6.7 7.0 301.0 57.9 4.7 8.1 765.0 19.2 40.2 5.8
75% 0.5 19.4 7.5 9.0 401.0 59.7 10.1 10.2 1017.0 21.4 43.3 9.8
90% 0.6 23.5 8.0 10.0 479.0 60.1 10.5 11.5 1090.0 23.9 45.7 10.3
Max 0.6 34.8 12.2 42.0 534.0 60.5 20.8 18.9 1215.0 26.9 50.0 18.1
Mean 0.4 17.7 7.0 8.6 313.3 57.9 6.0 8.6 807.8 19.1 41.2 7.0
50% 0.4 15.3 6.8 1.0 260.0 52.1 4.2 4.1 408.0 40.5 123.0 7.8
75% 0.5 21.6 7.4 12.0 380.0 54.2 9.5 5.2 550.0 54.2 129.6 11.5
90% 0.6 29.0 8.4 23.0 438.0 55.1 10.3 5.7 629.0 72.7 131.7 13.2
Max 0.7 144.0 15.2 62.0 646.0 56.2 20.5 7.0 784.0 91.7 143.2 20.4
Mean 0.4 21.4 7.1 8.8 303.7 51.7 5.8 4.3 435.9 44.9 123.7 9.4
50% 0.4 17.0 6.9 6.0 296.0 53.5 4.3 4.7 472.0 39.4 107.6 6.8
75% 0.5 20.3 7.9 13.0 371.0 55.4 9.0 5.8 556.0 49.3 112.0 12.8
90% 0.6 25.9 9.6 20.0 425.0 56.0 9.9 6.6 656.0 59.9 114.8 17.3
Max 0.7 33.2 15.4 42.0 515.0 57.0 20.4 8.9 671.0 72.8 127.5 25.6
Mean 0.4 17.7 7.4 9.2 301.7 53.7 5.8 5.0 481.1 42.5 108.7 9.1
50% 0.5 16.9 6.9 7.0 287.0 60.4 4.8 9.5 876.0 7.1 8.7 8.1
75% 0.6 21.1 7.6 11.0 422.0 61.4 10.0 12.5 1186.0 8.8 9.9 13.5
90% 0.6 27.4 8.1 32.0 488.0 62.2 10.7 13.5 1433.0 10.0 11.2 16.1
Max 0.7 117.0 13.2 52.0 542.0 63.0 21.6 17.2 1712.0 15.3 16.4 23.3
Mean 0.5 20.7 7.2 12.4 326.1 59.9 6.1 10.0 974.3 7.4 9.1 10.1
Table H.3: Workload characteristics and performance are measured for each sample, on each storage device. The minimum value, percentiles, maximum value and average are reported for each measurement. In addition, the relative difference between the average performance of the best and second-to-best iteration is reported. This value quantifies the change in a given measurement across multiple runs of the same sample on the same storage device.
107
APPENDIX H. CELLO MODEL TRAINING 1
20
Array A Array B Array C Array D
18 Bandwidth (MB/sec)
0.6 0.4
14 12 10 8 6 4 2 0 0
16
14
8
6
4
2
0
0
12
Array A Array B Array C Array D
0.2
10
Fraction
0.8
16
5
10
Bandwidth (MB/sec) 1
1800
Throughput (IO/sec)
0.4
1400
1600
5
10
80 Latency (ms)
0.6 0.4
30
35
40
70 60 50 40 30 20 10
80
0 70
60
40
30
20
10
50
Array A Array B Array C Array D
15 20 25 Sample number
Array A Array B Array C Array D
90
0
40
600
0
0.8
Fraction
35
800
100
Latency (ms)
30
1000
200
1
0
40
1200
Throughput (IO/sec)
0.2
35
400 1400
800
600
400
200
0
0
1200
Array A Array B Array C Array D
0.2
1000
Fraction
0.6
30
Array A Array B Array C Array D
1600 0.8
15 20 25 Sample number
0
5
10
15 20 25 Sample number
Figure H.1: The cumulative distribution of performance is shown (top). In addition, the performance of each array is shown (bottom), sorted by the performance of Array C.
108
Bandwidth (MB/sec) on Array A
Throughput (IO/sec) on Array A
0.14 Fraction
Fraction
0.13 0.12 0.11 0.1 0.09 0.08
<5
[5,
6)
[6,
8)
[8,
[9,
9)
10
)
[10
0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
0.25
[ [ [ [ [ [ > 44 544, 627, 711, 794, 878, 962, =10 4 62 7 7 8 9 1 7) 11) 94) 78) 62) 045 5 )
Fraction
Fraction
0.2
0.1 0.05
[2,
3)
[3,
4)
[4,
[4,
4)
5)
[5,
6)
[6,
6)
0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
6
0.15
0.3
0.2
0.25
0.15 0.1
0
4)
4)
[4,
[5,
5)
5)
[5,
6)
[6,
7)
<3
[ [ [ [ [ [ > 30 330, 377, 425, 472, 520, 568, =61 5 37 4 4 5 5 6 7) 25) 72) 20) 68) 15)
7
Bandwidth (MB/sec) on Array D
0.16 0.14 0.12 0.1
0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
8)
[8,
10
)
[10
[11 [12 >= 1 , , 1) 12) 13) 3
,1
0.1
<2
4
[24
[29 [35 [40 [46 [52 >= 5 , , , , , 9) 35) 40) 46) 52) 57) 7
,2
Latency (ms) on Array D 0.25 0.2
<5
35
[7,
0.2 0.15
0
Fraction
Fraction
Fraction
0.2 0.18
7)
[25 [34 [43 [52 [61 >= 7 , , , , , 5) 34) 43) 52) 61) 70) 0
,2
Throughput (IO/sec) on Array D
0.22
[6,
[16
0.05
>=
0.24
<6
<1
Latency (ms) on Array C
0.25
0.05
0.1
[4,
0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02
6
Fraction
Fraction
Fraction
0.2
[3,
[12 [14 [16 [18 [20 >= 2 , , , , , 2) 14) 16) 18) 20) 22) 2
,1
Throughput (IO/sec) on Array C
0.25
<3
[10
Latency (ms) on Array B
[ [ [ [ [ [ > 74 274, 329, 385, 440, 496, 552, =60 7 32 3 4 4 5 6 9) 85) 40) 96) 52) 07)
Bandwidth (MB/sec) on Array C
0.08
0
<2
>=
0.3
0.05
<1
Throughput (IO/sec) on Array B
0.15
<2
0.1
0
Fraction
Bandwidth (MB/sec) on Array B
0.2 0.15
0.05
<5
[11 >= 1 , 1) 12) 2
,1
0.25
0
Latency (ms) on Array A 0.3
Fraction
0.15
[53
[6 [8 [9 [1 [1 > 5, 70, 04, 38, 073 207 =134 , , 67 8 9 1 0) 04) 38) 073 120 134 2 ) 7) 2)
0.15 0.1 0.05 0
<5
Figure H.2: Probability distributions of performance.
[5,
6)
[6,
6)
[6,
7)
[7,
8)
[8,
9)
[9,
> 10 =10 )
109
APPENDIX H. CELLO MODEL TRAINING
Array A to Array B (min 0.33, max 0.73) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .37 0.37 0.41 0.45 0.48 0.52 0.56 =0.6 ,0 , , , , , .41 0.45 0.48 0.52 0.56 0.60 0 ) ) ) ) ) )
Array A to Array C (min 0.39, max 0.87) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08
<0
[ [ [ [ [ [ > .44 0.44 0.49 0.54 0.59 0.64 0.69 =0.7 ,0 , , , , , .49 0.54 0.59 0.64 0.69 0.74 4 ) ) ) ) ) )
Array B to Array A (min 1.37, max 3.00) 0.35 0.3 0.25 0.2 0.15 0.1 0.05
<1
[ [ [ [ [ [ > .55 1.55 1.73 1.91 2.09 2.27 2.46 =2.6 ,1 , , , , , .73 1.91 2.09 2.27 2.46 2.64 4 ) ) ) ) ) )
<1
[ [ [ [ [ [ > .00 1.00 1.06 1.13 1.19 1.25 1.32 =1.3 ,1 , , , , , .06 1.13 1.19 1.25 1.32 1.38 8 ) ) ) ) ) )
0.25 0.2 0.15 0.1 0.05
[ [ [ [ [ [ > .31 1.31 1.46 1.61 1.76 1.91 2.06 =2.2 ,1 , , , , , .46 1.61 1.76 1.91 2.06 2.21 1 ) ) ) ) ) )
0
<0
[ [ [ [ [ [ > .54 0.54 0.61 0.68 0.74 0.81 0.88 =0.9 ,0 , , , , , .61 0.68 0.74 0.81 0.88 0.95 5 ) ) ) ) ) )
Array D to Array A (min 0.62, max 1.15) 0.25
0.3
0.2
0.25 0.2
0.1
0.15
0.05 0
0.1
<0
.67
[0.
[ [ [ [ [ > 67 0.72 0.77 0.83 0.88 0.93 =0.9 ,0 , , , , , .72 0.77 0.83 0.88 0.93 0.98 8 ) ) ) ) ) )
Array B to Array D (min 1.26, max 3.53) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .49 1.49 1.72 1.95 2.18 2.41 2.64 =2.8 ,1 , , , , , .72 1.95 2.18 2.41 2.64 2.87 7 ) ) ) ) ) )
Array C to Array D (min 1.14, max 2.97) 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<1
[ [ [ [ [ [ > .34 1.34 1.54 1.74 1.94 2.13 2.33 =2.5 ,1 , , , , , .54 1.74 1.94 2.13 2.33 2.53 3 ) ) ) ) ) )
Array D to Array B (min 0.28, max 0.79) 0.35
0.15
[ [ [ [ [ [ > .94 0.94 1.01 1.08 1.15 1.22 1.30 =1.3 ,1 , , , , , .01 1.08 1.15 1.22 1.30 1.37 7 ) ) ) ) ) )
Array C to Array B (min 0.47, max 1.07) 0.3
<1
<0
Array B to Array C (min 0.94, max 2.11) 0.15 0.14 0.13 0.12 0.11 0.1 0.09 0.08
Array C to Array A (min 1.15, max 2.59) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
Array A to Array D (min 0.87, max 1.62) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
0.05
<0
[ [ [ [ [ [ > .33 0.33 0.37 0.42 0.46 0.50 0.55 =0.5 ,0 , , , , , .37 0.42 0.46 0.50 0.55 0.59 9 ) ) ) ) ) )
Array D to Array C (min 0.34, max 0.87) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .40 0.40 0.47 0.53 0.60 0.66 0.73 =0.7 ,0 , , , , , .47 0.53 0.60 0.66 0.73 0.79 9 ) ) ) ) ) )
Figure H.3: Each graph shows the probability distribution of the Bandwidth relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
110
Array A to Array B (min 0.39, max 0.73)
Array A to Array C (min 0.47, max 0.79)
Array A to Array D (min 0.76, max 1.52)
0.3
0.3
0.3
0.25
0.25
0.25
0.2
0.2
0.2
0.15
0.15
0.15
0.1
0.1
0.1
0.05
0.05
0.05
0
<0
[ [ [ [ [ [ > .42 0.42 0.45 0.48 0.52 0.55 0.58 =0.6 ,0 , , , , , .45 0.48 0.52 0.55 0.58 0.61 1 ) ) ) ) ) )
0
<0
[ [ [ [ [ [ > .51 0.51 0.54 0.57 0.61 0.64 0.67 =0.7 ,0 , , , , , .54 0.57 0.61 0.64 0.67 0.71 1 ) ) ) ) ) )
Array B to Array A (min 1.37, max 2.58) 0.3
0.25
0.25 0.2
0.15 0.15
0.1
0.1
0.05 0
<1
.49
[1.
[ [ [ [ [ > 49 1.61 1.73 1.84 1.96 2.08 =2.2 ,1 , , , , , .61 1.73 1.84 1.96 2.08 2.20 0 ) ) ) ) ) )
0.05
<0
[ [ [ [ [ [ > .93 0.93 0.99 1.06 1.13 1.20 1.27 =1.3 ,0 , , , , , .99 1.06 1.13 1.20 1.27 1.34 4 ) ) ) ) ) )
Array C to Array A (min 1.26, max 2.11) 0.3 0.25 0.2
<1
[ [ [ [ [ [ > .72 1.72 1.89 2.05 2.21 2.37 2.54 =2.7 ,1 , , , , , .89 2.05 2.21 2.37 2.54 2.70 0 ) ) ) ) ) )
Array C to Array B (min 0.58, max 1.17)
Array C to Array D (min 1.18, max 2.82) 0.3
0.2
0.25 0.2
0.15
0.15 0.1
0.1
0.1
0.05
0.05
<1
[ [ [ [ [ [ > .35 1.35 1.44 1.53 1.62 1.71 1.80 =1.8 ,1 , , , , , .44 1.53 1.62 1.71 1.80 1.89 9 ) ) ) ) ) )
0
0.05
<0
[ [ [ [ [ [ > .65 0.65 0.72 0.78 0.85 0.92 0.98 =1.0 ,0 , , , , , .72 0.78 0.85 0.92 0.98 1.05 5 ) ) ) ) ) )
Array D to Array A (min 0.66, max 1.31) 0.25 0.2 0.15 0.1 0.05 0
[ [ [ [ [ [ > .84 0.84 0.92 1.00 1.08 1.17 1.25 =1.3 ,0 , , , , , .92 1.00 1.08 1.17 1.25 1.33 3 ) ) ) ) ) )
Array B to Array D (min 1.56, max 3.11) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
0.25
0.15
0
<0
Array B to Array C (min 0.86, max 1.72)
0.3 0.2
0
<0
[ [ [ [ [ [ > .71 0.71 0.76 0.82 0.87 0.92 0.97 =1.0 ,0 , , , , , .76 0.82 0.87 0.92 0.97 1.03 3 ) ) ) ) ) )
0
<1
[ [ [ [ [ [ > .36 1.36 1.53 1.71 1.88 2.06 2.23 =2.4 ,1 , , , , , .53 1.71 1.88 2.06 2.23 2.41 1 ) ) ) ) ) )
Array D to Array B (min 0.32, max 0.64) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<0
[ [ [ [ [ [ > .35 0.35 0.39 0.42 0.45 0.48 0.51 =0.5 ,0 , , , , , .39 0.42 0.45 0.48 0.51 0.54 4 ) ) ) ) ) )
Array D to Array C (min 0.35, max 0.85) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .41 0.41 0.46 0.51 0.56 0.61 0.66 =0.7 ,0 , , , , , .46 0.51 0.56 0.61 0.66 0.71 1 ) ) ) ) ) )
Figure H.4: Each graph shows the probability distribution of the Throughput relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
111
APPENDIX H. CELLO MODEL TRAINING
Array A to Array B (min 0.57, max 3.72) 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .93 0.93 1.28 1.64 1.99 2.34 2.70 =3.0 ,1 , , , , , .28 1.64 1.99 2.34 2.70 3.05 5 ) ) ) ) ) )
Array A to Array C (min 1.19, max 4.77) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
0.25 0.2 0.15 0.1
<1
[ [ [ [ [ [ > .42 1.42 1.65 1.88 2.12 2.35 2.58 =2.8 ,1 , , , , , .65 1.88 2.12 2.35 2.58 2.82 2 ) ) ) ) ) )
Array B to Array A (min 0.27, max 1.75) 0.3
0.25
0.25
0.2
0.2
0.15
0.15
0.1
0.1
0.05
0.05
<0
[ [ [ [ [ [ > .34 0.34 0.42 0.50 0.57 0.65 0.72 =0.8 ,0 , , , , , .42 0.50 0.57 0.65 0.72 0.80 0 ) ) ) ) ) )
[ [ [ [ [ [ > .26 0.26 0.30 0.35 0.40 0.44 0.49 =0.5 ,0 , , , , , .30 0.35 0.40 0.44 0.49 0.54 4 ) ) ) ) ) )
0
Array B to Array D (min 0.08, max 0.68)
0.2 0.15 0.1 0.05
<0
[ [ [ [ [ [ > .65 0.65 0.79 0.93 1.08 1.22 1.36 =1.5 ,0 , , , , , .79 0.93 1.08 1.22 1.36 1.51 1 ) ) ) ) ) )
0
<0
[ [ [ [ [ [ > .12 0.12 0.16 0.20 0.24 0.28 0.32 =0.3 ,0 , , , , , .16 0.20 0.24 0.28 0.32 0.36 6 ) ) ) ) ) )
Array C to Array B (min 0.22, max 1.98)
Array C to Array D (min 0.09, max 0.49)
0.25
0.3
0.2
0.25 0.2
0.15
0.15 0.1
0.1
0.05
<0
[ [ [ [ [ [ > .28 0.28 0.36 0.43 0.50 0.58 0.65 =0.7 ,0 , , , , , .36 0.43 0.50 0.58 0.65 0.72 2 ) ) ) ) ) )
0
Array D to Array A (min 1.14, max 4.78) 0.3 0.2 0.15 0.1 0.05
<1
[ [ [ [ [ [ > .48 1.48 1.83 2.17 2.51 2.86 3.20 =3.5 ,1 , , , , , .83 2.17 2.51 2.86 3.20 3.55 5 ) ) ) ) ) )
0.05
<0
[ [ [ [ [ [ > .37 0.37 0.52 0.67 0.82 0.97 1.12 =1.2 ,0 , , , , , .52 0.67 0.82 0.97 1.12 1.27 7 ) ) ) ) ) )
0
Array D to Array B (min 1.46, max 13.07) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
0.25
0
<0
0.25
Array C to Array A (min 0.21, max 0.84) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.05
Array B to Array C (min 0.51, max 4.63)
0.3
0
Array A to Array D (min 0.21, max 0.88) 0.3
<2
[ [ [ [ [ [ > .80 2.80 4.13 5.47 6.80 8.14 9.47 =10 ,4 , , , , , . .13 5.47 6.80 8.14 9.47 10.8 81 ) ) ) ) ) 1)
<0
[ [ [ [ [ [ > .12 0.12 0.15 0.18 0.21 0.25 0.28 =0.3 ,0 , , , , , .15 0.18 0.21 0.25 0.28 0.31 1 ) ) ) ) ) )
Array D to Array C (min 2.04, max 11.05) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08
<2
[ [ [ [ [ [ > .99 2.99 3.95 4.91 5.86 6.82 7.78 =8.7 ,3 , , , , , .95 4.91 5.86 6.82 7.78 8.73 3 ) ) ) ) ) )
Figure H.5: Each graph shows the probability distribution of the Latency relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
1
0.8
0.8
0.6
0.6
45 40
25
20
40
Write fraction
15
10
0
5
0
0.7
0.6
0.4
0.3
0.2
0.1
0
0
Array A Array B Array C Array D
0.2
35
Array A Array B Array C Array D
0.2
0.4
35
0.4
30
Fraction
1
0.5
Fraction
112
Write size (KB)
1
1 0.9
0.8
0.8
Fraction
0.6 0.4
0.6 0.5 0.4 0.3
Write queue
Write jump (MB)
Figure H.6: The cumulative distribution of workload characteristics.
30
20
15
10
5
0
0
25
0.1 70
40
30
20
10
0
Array A Array B Array C Array D
0.2
60
0.2
50
Array A Array B Array C Array D 0
Fraction
0.7
113
APPENDIX H. CELLO MODEL TRAINING
Overall Predictor Read queue Write fraction Write size Write queue
Score 1.00 0.64 0.17 0.07
Overall Predictor Read queue Write fraction Write size Read jump Write queue
Score 1.00 0.20 0.18 0.16 0.14
Overall Predictor Throughput Bandwidth Read queue Read latency Latency Write size Read jump Write queue
Score 1.00 0.97 0.72 0.33 0.30 0.24 0.18 0.15
Overall Predictor Write fraction Throughput Read queue Latency Write size Read latency Write queue Write latency
Score 1.00 0.79 0.71 0.71 0.66 0.55 0.50 0.40
Absolute Bandwidth Throughput Predictor Score Predictor Score Read queue 1.00 Read queue 1.00 Write fraction 0.40 Write fraction 0.17 Write size 0.28 Write size 0.10 Write jump 0.16 Read jump 0.10 Relative Bandwidth Throughput Predictor Score Predictor Score Read queue 1.00 Read queue 1.00 Write size 0.20 Read jump 0.11 Read size 0.16 Write queue 0.06 Write fraction 0.09 Write size 0.04 Read jump 0.05 Read size 0.02 Relative Performance Bandwidth Throughput Predictor Score Predictor Score Bandwidth 1.00 Throughput 1.00 Throughput 0.35 Read queue 0.74 Read queue 0.30 Read jump 0.10 Read latency 0.15 Write queue 0.08 Read size 0.09 Read size 0.07 Write fraction 0.08 Bandwidth 0.05 Write size 0.04 Write jump 0.03 Latency 0.04 Write fraction 0.03 Relative Fitness Bandwidth Throughput Predictor Score Predictor Score Write size 1.00 Read latency 1.00 Throughput 0.60 Throughput 0.98 Read queue 0.57 Write latency 0.77 Write fraction 0.48 Read queue 0.47 Read size 0.47 Bandwidth 0.23 Write queue 0.35 Write fraction 0.23 Bandwidth 0.19 Write jump 0.20 Read jump 0.17 Latency 0.18
Latency Predictor Write fraction Read queue Write queue Read size
Score 1.00 0.57 0.13 0.09
Latency Predictor Read queue Write fraction Write queue Write size Read jump
Score 1.00 0.93 0.64 0.59 0.50
Latency Predictor Bandwidth Latency Read latency Write size Throughput Read jump Write queue Write fraction
Score 1.00 0.90 0.77 0.72 0.41 0.36 0.27 0.21
Latency Predictor Write fraction Latency Write queue Read jump Read queue Read latency Bandwidth Read size
Score 1.00 0.89 0.47 0.30 0.25 0.16 0.13 0.11
Table H.4: The normalized importance measure of each predictor.
Model Absolute Relative Relative Performance Relative Fitness
Overall 5.6 3.2 4.2 3.2
Absolute Array Array Array Array Relative Array Array Array Array Relative Performance Array Array Array Array Relative Fitness Array Array Array Array
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Pairwise Array 4.7 Array 4.7 2.3 2.7 2.7 Array 7.0 5.0 3.3 4.0 Array 1 3.3 4.7 2.3
Bandwidth B C 6.0 7.0 Bandwidth A B C 2.0 2.0 5.0 2.0 6.0 7.0 4.0 3.0 7.0 3.0 2.0 4.0 Bandwidth A B C 6.0 2.0 3.0 3.0 7.0 4.0 6.0 7.0 6.0 7.0 2.0 3.0 Bandwidth A B C 1 9.0 2.0 6.0 1 9.0 11.0 2.0 1 3.0 3.0 3.0 A 2.0 -
Bandwidth 5.5 3.2 4.3 4.8
D 7.0 D 2.0 2.0 2.0 7.0 D 8.0 3.0 4.0 7.0 D 2.0 5.0 3.0 1
Array 6.3 Array 2.0 6.3 5.3 2.7 Array 3.3 6.7 6.7 5.0 Array 4.7 1 2.0 2.3
Throughput 4.5 3.9 4.5 1.9
Array 5.0 Array 3.0 5.7 5.0 2.7 Array 3.7 3.7 5.7 2.3 Array 2.0 4.0 1 2.3
Latency 6.8 2.5 3.8 2.8
Array 6.3 Array 4.7 2.3 2.3 6.3 Array 5.3 4.3 4.0 7.0 Array 4.0 3.7 3.0 1
Throughput B C D 7.0 3.0 3.0 Throughput A B C D 5.0 3.0 3.0 5.0 3.0 7.0 7.0 3.0 3.0 7.0 3.0 3.0 4.0 3.0 3.0 3.0 Throughput A B C D 8.0 7.0 7.0 3.0 3.0 6.0 3.0 3.0 3.0 7.0 6.0 5.0 3.0 7.0 3.0 6.0 Throughput A B C D 1 3.0 1.0 2.0 2.0 1 2.0 2.0 1.0 2.0 1 2.0 1.0 3.0 2.0 1 A 5.0 -
Table H.5: Tree sizes (leaf nodes) and their averages.
A 7.0 A 7.0 2.0 1.0 1.0 A 7.0 9.0 1.0 2.0 A 1 2.0 2.0 3.0
Latency B C 6.0 5.0 Latency B C 1.0 1.0 6.0 3.0 6.0 5.0 3.0 1.0 Latency B C 1.0 1.0 7.0 4.0 6.0 5.0 6.0 1.0 Latency B C 2.0 3.0 1 1.0 2.0 1 1.0 2.0
D 9.0 D 7.0 2.0 2.0 9.0 D 5.0 7.0 3.0 8.0 D 8.0 4.0 4.0 1
115
APPENDIX H. CELLO MODEL TRAINING
Bandwidth error (Cello) Median relative error (Avg.)
Median relative error (Avg.)
Overall error (Cello) 0.18 0.17 0.16 0.15 0.14 0.13 0.12 0.11 0.1 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
0.22 0.2 0.18 0.16 0.14 0.12 0.1
70
30 AM (i=j) AM
RF
0.115 0.11 0.105 0.1 0.095 0.09 0.085 0.08 0.075 0.07 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
RM RM’
70 RF
Latency error (Cello) Median relative error (Avg.)
Median relative error (Avg.)
Throughput error (Cello)
40 50 60 Samples used for training (%)
70 RF
0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
Figure H.7: Error vs. training. The training set size varies from 25% of the collected samples to 75%; the remaining samples are used for testing. For each training set size, the Median relative error is calculated for each pairing of devices. They are then averaged to produce the values shown.
116
Bandwidth error (Cello) Mean relative error (Avg.)
Mean relative error (Avg.)
Overall error (Cello) 0.28 0.27 0.26 0.25 0.24 0.23 0.22 0.21 0.2 0.19 0.18 0.17 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
0.28 0.26 0.24 0.22 0.2 0.18 0.16 0.14
70
30
RF
AM (i=j) AM
30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
RM RM’
70 RF
Latency error (Cello)
0.19 0.18 0.17 0.16 0.15 0.14 0.13 0.12 0.11 0.1
Mean relative error (Avg.)
Mean relative error (Avg.)
Throughput error (Cello)
40 50 60 Samples used for training (%)
70 RF
0.45 0.4 0.35 0.3 0.25 0.2 0.15 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
Figure H.8: Error vs. training. The training set size varies from 25% of the collected samples to 75%; the remaining samples are used for testing. For each training set size, the Mean relative error is calculated for each pairing of devices. They are then averaged to produce the values shown.
Appendix I
TPC-C model training
Application TPC-C (tpcc) Total used
Samples 50
Iters 3
First sample 0 25
Last sample 24
Table I.1: Multiple training samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Array A 0.18 8.52 8.14 125.28 273.16 18.96 1.00 1.69 211.60 5.78 20.82 4.15
Array B 0.21 8.45 8.12 208.12 270.48 30.82 1.00 1.12 140.04 17.29 85.17 7.31
Array C 0.16 8.61 8.17 101.56 277.00 13.28 1.00 1.96 243.84 5.12 26.93 2.57
Array D 0.24 8.77 8.12 132.96 299.40 13.52 1.00 2.40 296.92 1.94 2.81 2.15
MAD 0.0 0.1 0.0 33.1 9.7 5.8 0.0 0.4 47.3 4.9 25.6 1.7
COV 14.1% 1.4% 0.2% 28.1% 4.1% 37.2% 0.1% 25.9% 25.5% 77.2% 91.0% 50.1%
Max. Diff. 50.0% 3.8% 0.6% 104.9% 10.7% 132.1% 0.0% 114.3% 112.0% 791.2% 2931.0% 240.0%
Table I.2: Workload characteristics and performance are measured for each sample, on each storage device. The average value for each measurement is reported in this table. The mean absolute deviation (MAD), coefficient of variation (COV), and maximum relative differences of these averages are also reported; these metrics quantify how the averages change among the storage devices. For example, the mean average deviation of the averages for Read latency (ms) is 1.68, their coeefficient of variation is 50.144%, and the maximum relative difference is 240.00%.
117
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 2.6% 0.3% 0.2% 15.7% 0.2% 10.1% 0.0% 1.4% 1.2% 2.0% 7.8% 1.1%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 12.6% 0.0% 0.1% 7.4% 1.0% 9.3% 0.1% 2.8% 2.4% 14.3% 5.4% 0.4%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 1.5% 0.2% 0.0% 14.0% 0.8% 14.9% 0.1% 1.7% 1.6% 6.4% 9.0% 1.5%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 3.1% 0.2% 0.1% 27.5% 0.3% 24.5% 0.0% 2.1% 2.0% 0.4% 5.7% 0.8%
Array A Min 25% 0.0 0.1 7.6 8.2 8.0 8.0 54.0 86.0 182.0 210.0 4.1 12.7 1.0 1.0 1.3 1.4 170.0 180.0 3.0 5.2 14.8 15.6 2.9 3.3 Array B Min 25% 0.0 0.2 7.4 8.3 8.0 8.0 58.0 101.0 183.0 218.0 4.8 25.8 1.0 1.0 0.9 1.0 114.0 125.0 5.6 13.2 35.3 73.8 5.5 6.6 Array C Min 25% 0.1 0.1 7.7 8.4 8.0 8.0 44.0 63.0 175.0 242.0 8.5 9.4 1.0 1.0 1.3 1.8 166.0 211.0 3.5 4.3 18.6 19.7 1.8 2.2 Array D Min 25% 0.1 0.2 7.8 8.4 8.0 8.0 64.0 86.0 184.0 287.0 3.8 7.8 1.0 1.0 0.9 2.2 110.0 278.0 1.5 1.6 0.7 1.2 1.6 1.9
50% 0.1 8.5 8.1 97.0 277.0 13.7 1.0 1.6 194.0 6.1 16.6 4.2
75% 0.2 8.7 8.2 131.0 311.0 17.5 1.0 1.8 226.0 6.4 19.0 4.6
90% 0.2 8.8 8.3 152.0 324.0 33.0 1.0 2.2 275.0 6.8 34.0 4.9
Max 1.0 9.3 8.4 542.0 333.0 48.5 1.0 2.3 296.0 7.0 40.6 5.1
Mean 0.2 8.5 8.1 125.3 273.2 19.0 1.0 1.7 211.6 5.8 20.8 4.1
50% 0.2 8.4 8.1 163.0 272.0 31.6 1.0 1.1 132.0 17.9 82.7 7.2
75% 0.3 8.5 8.2 194.0 302.0 35.6 1.0 1.2 157.0 20.7 97.3 8.0
90% 0.3 8.6 8.3 241.0 319.0 41.0 1.0 1.3 159.0 21.9 106.5 8.4
Max 0.3 10.1 8.8 1157.0 381.0 43.4 1.0 1.4 180.0 24.5 112.6 8.6
Mean 0.2 8.5 8.1 208.1 270.5 30.8 1.0 1.1 140.0 17.3 85.2 7.3
50% 0.2 8.7 8.1 96.0 292.0 10.1 1.0 2.0 246.0 4.9 21.5 2.5
75% 0.2 8.8 8.3 126.0 306.0 13.3 1.0 2.1 266.0 5.8 25.7 2.7
90% 0.2 8.9 8.3 137.0 316.0 17.1 1.0 2.2 274.0 6.1 40.9 3.2
Max 0.2 9.0 8.4 244.0 340.0 34.9 1.0 2.6 311.0 8.1 67.3 3.8
Mean 0.2 8.6 8.2 101.6 277.0 13.3 1.0 2.0 243.8 5.1 26.9 2.6
50% 0.2 8.8 8.1 123.0 319.0 8.5 1.0 2.5 306.0 1.7 1.7 2.0
75% 0.2 9.1 8.2 145.0 322.0 10.8 1.0 2.6 320.0 2.0 2.9 2.2
90% 0.2 9.2 8.2 191.0 327.0 13.4 1.0 2.8 338.0 2.5 5.5 2.6
Max 1.0 9.4 8.3 289.0 362.0 61.3 1.0 2.9 364.0 3.4 11.9 3.4
Mean 0.2 8.8 8.1 133.0 299.4 13.5 1.0 2.4 296.9 1.9 2.8 2.1
Table I.3: Workload characteristics and performance are measured for each sample, on each storage device. The minimum value, percentiles, maximum value and average are reported for each measurement. In addition, the relative difference between the average performance of the best and second-to-best iteration is reported. This value quantifies the change in a given measurement across multiple runs of the same sample on the same storage device.
119 3
0.8
2.5
Bandwidth (MB/sec)
1
0.6 0.4 0.2
2
1.5
1
0.5
0
0
Array A Array B Array C Array D
2 1.5 1 0.5
3
Array A Array B Array C Array D 2.5
Fraction
APPENDIX I. TPC-C MODEL TRAINING
0
5
400
0.8
350 Throughput (IO/sec)
1
0.6 0.4 Array A Array B Array C Array D
0
Latency (ms)
0.6 0.4 Array A Array B Array C Array D
5
10 15 Sample number
20
25
Array A Array B Array C Array D
15 10 5 0
25
15
10
5
25
100
20
0
20
200
0.8
20
Fraction
10 15 Sample number
250
25
Latency (ms)
25
300
1
0
20
Array A Array B Array C Array D
Throughput (IO/sec)
0.2
10 15 Sample number
150 350
200
150
100
50
0
0
300
0.2
250
Fraction
Bandwidth (MB/sec)
0
5
Figure I.1: The cumulative distribution of performance is shown (top). In addition, the performance of each array is shown (bottom), sorted by the performance of Array C.
120
Bandwidth (MB/sec) on Array A
Throughput (IO/sec) on Array A
Fraction
0.2 Fraction
Latency (ms) on Array A
0.3
0.15 0.1
0.35
0.25
0.3
0.2
0.25 Fraction
0.25
0.15 0.1 0.05
0.05 0
0
<1
[1,
2)
[2,
2)
[2,
2)
[2,
2)
[2,
2)
[2,
2)
0.2 0.15 0.1 0.05
<1
[ [ [ [ [ [ > 84 184, 197, 211, 224, 238, 252, =26 5 19 2 2 2 2 2 7) 11) 24) 38) 52) 65)
>=
2
Bandwidth (MB/sec) on Array B
0
<3
[3,
Throughput (IO/sec) on Array B
0.3 0.25
[4,
4)
[4,
4)
5)
[5,
5)
[5,
6)
[6,
6)
>=
6
Latency (ms) on Array B
0.35
0.3
0.3
0.25
0.2 0.15
Fraction
Fraction
Fraction
0.25 0.2 0.15 0.1 0.1
0.05 0
0.05 0
<1
[1,
1)
[1,
1)
[1,
1)
[1,
1)
[1,
1)
[1,
1)
<1
[ [ [ [ [ [ > 20 120, 126, 132, 138, 144, 150, =15 6 12 1 1 1 1 1 6) 32) 38) 44) 50) 56)
1
0.25 0.2
0.2 0.15
0.05
0
[2,
2)
[2,
2)
[2,
2)
[2,
2)
Bandwidth (MB/sec) on Array D
0.14 0.12
[1 [2 [2 [2 [2 > 2, 98, 14, 30, 46, 62, =278 19 2 2 2 2 2 8) 14) 30) 46) 62) 78)
0.06 0.04
0.3
0.4
0.25
0.3 0.2 0.1
0.1
0
1)
[1,
2)
[2,
2)
[2,
2)
[2,
2)
[2,
3)
>=
3
[4,
5)
[5,
5)
[5,
6)
[6,
6)
[6,
6)
>=
6
0.2 0.15 0.1
<1
39
[1,
4)
0.35
Fraction
0.5
<1
[4,
Latency (ms) on Array D
0.5
0
<4
Throughput (IO/sec) on Array D 0.6
Fraction
Fraction
0.16
0.6
0.2
[12 [14 [16 [19 >= 2 , , , , 2) 14) 16) 19) 21) 1
,1
0.1
[18
2
0.3
[10
0.08
<1
>=
0.4
)
0.2
82
2)
10
0.18
0.1 0.05
[2,
[8,
Latency (ms) on Array C
0.15
0.1
2)
<8
0.22
Fraction
0.3 0.25
[1,
0
Throughput (IO/sec) on Array C 0.3
Fraction
Fraction
Bandwidth (MB/sec) on Array C
<1
0.1 0.05
>=
0.35
0
0.2 0.15
[13
9,
[16
16
8)
8,
[19
19
7)
7,
[22
22
6)
6,
[25
25
4)
4,
[28
28
3)
3,
>=
31 2 2)
31
0.05 0
Figure I.2: Probability distributions of performance.
<2
[2,
2)
[2,
2)
[2,
2)
[2,
2)
[2,
2)
[2,
2)
>=
2
121
APPENDIX I. TPC-C MODEL TRAINING
Array A to Array B (min 0.56, max 0.77) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
Array A to Array C (min 0.60, max 1.55) 0.3 0.25 0.2 0.15 0.1 0.05
<0
[ [ [ [ [ [ > .58 0.58 0.61 0.63 0.65 0.68 0.70 =0.7 ,0 , , , , , .61 0.63 0.65 0.68 0.70 0.73 3 ) ) ) ) ) )
0
<0
[ [ [ [ [ [ > .71 0.71 0.82 0.92 1.03 1.14 1.25 =1.3 ,0 , , , , , .82 0.92 1.03 1.14 1.25 1.36 6 ) ) ) ) ) )
Array B to Array A (min 1.30, max 1.79) 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
0.2 0.15 0.1 0.05
[ [ [ [ [ [ > .35 1.35 1.41 1.46 1.52 1.57 1.63 =1.6 ,1 , , , , , .41 1.46 1.52 1.57 1.63 1.68 8 ) ) ) ) ) )
0
<1
[ [ [ [ [ [ > .12 1.12 1.28 1.43 1.58 1.73 1.89 =2.0 ,1 , , , , , .28 1.43 1.58 1.73 1.89 2.04 4 ) ) ) ) ) )
Array C to Array A (min 0.64, max 1.68) 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .69 0.69 0.74 0.78 0.83 0.87 0.92 =0.9 ,0 , , , , , .74 0.78 0.83 0.87 0.92 0.96 6 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .59 0.59 0.64 0.70 0.76 0.82 0.88 =0.9 ,0 , , , , , .64 0.70 0.76 0.82 0.88 0.93 3 ) ) ) ) ) )
[ [ [ [ [ [ > .57 0.57 0.75 0.93 1.11 1.29 1.47 =1.6 ,0 , , , , , .75 0.93 1.11 1.29 1.47 1.65 5 ) ) ) ) ) )
Array B to Array D (min 0.64, max 2.87) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .90 0.90 1.16 1.41 1.67 1.93 2.19 =2.4 ,1 , , , , , .16 1.41 1.67 1.93 2.19 2.44 4 ) ) ) ) ) )
Array C to Array B (min 0.43, max 1.03) 0.16 0.15 0.14 0.13 0.12 0.11 0.1 0.09 0.08
<0
[ [ [ [ [ [ > .46 0.46 0.50 0.53 0.57 0.60 0.64 =0.6 ,0 , , , , , .50 0.53 0.57 0.60 0.64 0.67 7 ) ) ) ) ) )
Array D to Array A (min 0.53, max 2.53) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
Array B to Array C (min 0.97, max 2.34) 0.25
<1
Array A to Array D (min 0.39, max 1.90) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array D (min 0.66, max 1.49) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .76 0.76 0.85 0.95 1.04 1.14 1.24 =1.3 ,0 , , , , , .85 0.95 1.04 1.14 1.24 1.33 3 ) ) ) ) ) )
Array D to Array B (min 0.35, max 1.56) 0.16 0.15 0.14 0.13 0.12 0.11 0.1 0.09 0.08
Array D to Array C (min 0.67, max 1.51) 0.3 0.25 0.2 0.15 0.1 0.05
<0
[ [ [ [ [ [ > .37 0.37 0.40 0.43 0.45 0.48 0.50 =0.5 ,0 , , , , , .40 0.43 0.45 0.48 0.50 0.53 3 ) ) ) ) ) )
0
<0
[ [ [ [ [ [ > .70 0.70 0.73 0.76 0.80 0.83 0.86 =0.8 ,0 , , , , , .73 0.76 0.80 0.83 0.86 0.89 9 ) ) ) ) ) )
Figure I.3: Each graph shows the probability distribution of the Bandwidth relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
122
Array A to Array B (min 0.57, max 0.78) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<0
[ [ [ [ [ [ > .59 0.59 0.61 0.63 0.65 0.67 0.70 =0.7 ,0 , , , , , .61 0.63 0.65 0.67 0.70 0.72 2 ) ) ) ) ) )
Array A to Array C (min 0.59, max 1.53) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .70 0.70 0.81 0.92 1.03 1.14 1.25 =1.3 ,0 , , , , , .81 0.92 1.03 1.14 1.25 1.36 6 ) ) ) ) ) )
Array B to Array A (min 1.28, max 1.75) 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
0.25 0.2 0.15 0.1 0.05
[ [ [ [ [ [ > .34 1.34 1.39 1.44 1.50 1.55 1.60 =1.6 ,1 , , , , , .39 1.44 1.50 1.55 1.60 1.66 6 ) ) ) ) ) )
0
<1
[ [ [ [ [ [ > .12 1.12 1.27 1.42 1.57 1.72 1.87 =2.0 ,1 , , , , , .27 1.42 1.57 1.72 1.87 2.02 2 ) ) ) ) ) )
Array C to Array A (min 0.65, max 1.69) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08
0.2 0.15 0.1 0.05
[ [ [ [ [ [ > .70 0.70 0.74 0.79 0.83 0.87 0.92 =0.9 ,0 , , , , , .74 0.79 0.83 0.87 0.92 0.96 6 ) ) ) ) ) )
0
<0
[ [ [ [ [ [ > .47 0.47 0.50 0.54 0.58 0.61 0.65 =0.6 ,0 , , , , , .50 0.54 0.58 0.61 0.65 0.68 8 ) ) ) ) ) )
Array D to Array A (min 0.54, max 2.55) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .60 0.60 0.65 0.71 0.77 0.82 0.88 =0.9 ,0 , , , , , .65 0.71 0.77 0.82 0.88 0.93 3 ) ) ) ) ) )
[ [ [ [ [ [ > .57 0.57 0.75 0.93 1.12 1.30 1.48 =1.6 ,0 , , , , , .75 0.93 1.12 1.30 1.48 1.66 6 ) ) ) ) ) )
Array B to Array D (min 0.64, max 2.79) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .89 0.89 1.13 1.38 1.63 1.88 2.12 =2.3 ,1 , , , , , .13 1.38 1.63 1.88 2.12 2.37 7 ) ) ) ) ) )
Array C to Array B (min 0.43, max 1.04) 0.25
<0
<0
Array B to Array C (min 0.97, max 2.31) 0.3
<1
Array A to Array D (min 0.39, max 1.85) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array D (min 0.66, max 1.52) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .76 0.76 0.85 0.95 1.04 1.14 1.23 =1.3 ,0 , , , , , .85 0.95 1.04 1.14 1.23 1.32 2 ) ) ) ) ) )
Array D to Array B (min 0.36, max 1.56) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<0
[ [ [ [ [ [ > .38 0.38 0.41 0.44 0.46 0.49 0.51 =0.5 ,0 , , , , , .41 0.44 0.46 0.49 0.51 0.54 4 ) ) ) ) ) )
Array D to Array C (min 0.66, max 1.51) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<0
[ [ [ [ [ [ > .69 0.69 0.73 0.76 0.79 0.82 0.86 =0.8 ,0 , , , , , .73 0.76 0.79 0.82 0.86 0.89 9 ) ) ) ) ) )
Figure I.4: Each graph shows the probability distribution of the Throughput relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
123
APPENDIX I. TPC-C MODEL TRAINING
Array A to Array B (min 1.60, max 6.08) 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .90 1.90 2.21 2.52 2.83 3.14 3.44 =3.7 ,2 , , , , , .21 2.52 2.83 3.14 3.44 3.75 5 ) ) ) ) ) )
Array A to Array C (min 0.59, max 2.19) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08
<0
[ [ [ [ [ [ > .66 0.66 0.74 0.81 0.89 0.96 1.04 =1.1 ,0 , , , , , .74 0.81 0.89 0.96 1.04 1.11 1 ) ) ) ) ) )
Array B to Array A (min 0.16, max 0.63) 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .22 0.22 0.28 0.33 0.39 0.45 0.50 =0.5 ,0 , , , , , .28 0.33 0.39 0.45 0.50 0.56 6 ) ) ) ) ) )
0.25
0.25
0.2
0.2
0.15
0.15
0.1
0.1
0.05
0.05 0
Array D to Array A (min 0.88, max 4.47) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.25 0.2 0.15 0.1 0.05
<0
[ [ [ [ [ [ > .21 0.21 0.25 0.30 0.34 0.38 0.43 =0.4 ,0 , , , , , .25 0.30 0.34 0.38 0.43 0.47 7 ) ) ) ) ) )
[ [ [ [ [ [ > .30 1.30 1.72 2.13 2.55 2.96 3.38 =3.8 ,1 , , , , , .72 2.13 2.55 2.96 3.38 3.80 0 ) ) ) ) ) )
0
<0
[ [ [ [ [ [ > .08 0.08 0.10 0.11 0.13 0.14 0.16 =0.1 ,0 , , , , , .10 0.11 0.13 0.14 0.16 0.17 7 ) ) ) ) ) )
<1
[ [ [ [ [ [ > .23 1.23 1.73 2.24 2.74 3.24 3.75 =4.2 ,1 , , , , , .73 2.24 2.74 3.24 3.75 4.25 5 ) ) ) ) ) )
Array C to Array D (min 0.26, max 0.95) 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<0
[ [ [ [ [ [ > .28 0.28 0.31 0.34 0.37 0.40 0.43 =0.4 ,0 , , , , , .31 0.34 0.37 0.40 0.43 0.46 6 ) ) ) ) ) )
Array D to Array B (min 1.66, max 15.58) 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
Array B to Array D (min 0.06, max 0.60)
Array C to Array B (min 0.73, max 6.11) 0.3
[ [ [ [ [ [ > .60 0.60 0.74 0.88 1.02 1.17 1.31 =1.4 ,0 , , , , , .74 0.88 1.02 1.17 1.31 1.45 5 ) ) ) ) ) )
[ [ [ [ [ [ > .28 0.28 0.33 0.38 0.43 0.48 0.53 =0.5 ,0 , , , , , .33 0.38 0.43 0.48 0.53 0.58 8 ) ) ) ) ) )
0.3
Array C to Array A (min 0.46, max 1.70)
<0
<0
Array B to Array C (min 0.16, max 1.37) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.3
0
Array A to Array D (min 0.22, max 1.13) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array D to Array C (min 1.05, max 3.91) 0.3 0.25 0.2 0.15 0.1 0.05
<3
.23
[3.
[ [ [ [ [ > 23 4.80 6.38 7.95 9.52 11.0 =12 ,4 , , , , 9 . .80 6.38 7.95 9.52 11.0 , 12 67 .67 ) ) ) ) 9) )
0
<1
[ [ [ [ [ [ > .37 1.37 1.69 2.01 2.33 2.65 2.97 =3.2 ,1 , , , , , .69 2.01 2.33 2.65 2.97 3.29 9 ) ) ) ) ) )
Figure I.5: Each graph shows the probability distribution of the Latency relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
1
0.8
0.8
0.6
0.6
0.8
0.8
0.6
0.6
10
6 200
150
100
45
40
35
30
25
20
15
5
10
Write queue
50
0 0
0
Array A Array B Array C Array D
0.2
Write jump (MB)
Figure I.6: The cumulative distribution of workload characteristics.
350
Array A Array B Array C Array D
0.4
300
0.4
250
Fraction
1
0.2
5
Write size (KB)
1
0
Fraction
Write fraction
4
3
2
1
0 0
0.2
0.15
0.1
0.05
0
0
Array A Array B Array C Array D
0.2 0.3
0.2
0.4
9
Array A Array B Array C Array D
8
0.4
7
Fraction
1
0.25
Fraction
124
125
APPENDIX I. TPC-C MODEL TRAINING
Overall Predictor Read jump Write queue Write fraction
Score 1.00 0.54 0.32
Overall Predictor Read jump Write size Write queue Write fraction Write jump Read size
Score 1.00 0.77 0.58 0.52 0.40 0.39
Overall Predictor Read jump Write size Bandwidth Write fraction Write queue Read latency Read size Write jump Latency Write latency
Score 1.00 0.82 0.79 0.46 0.44 0.44 0.37 0.32 0.12 0.12
Overall Predictor Write fraction Read latency Latency Write size Bandwidth Read jump Write queue Write latency Read size
Score 1.00 0.92 0.84 0.80 0.40 0.36 0.32 0.31 0.23
Absolute Bandwidth Throughput Predictor Score Predictor Score Read jump 1.00 Read jump 1.00 Read size 0.33 Write size 0.20 Write size 0.27 Write queue 0.19 Relative Bandwidth Throughput Predictor Score Predictor Score Read jump 1.00 Read jump 1.00 Write size 0.50 Write size 0.64 Read size 0.45 Write fraction 0.32 Write queue 0.30 Write queue 0.22 Write fraction 0.19 Read size 0.20 Write jump 0.08 Write jump 0.08 Relative Performance Bandwidth Throughput Predictor Score Predictor Score Read jump 1.00 Read jump 1.00 Bandwidth 0.69 Bandwidth 0.85 Write size 0.63 Write size 0.60 Read latency 0.26 Read latency 0.43 Write fraction 0.26 Write fraction 0.29 Read size 0.25 Read size 0.28 Write queue 0.19 Write jump 0.14 Write latency 0.15 Write latency 0.12 Latency 0.13 Throughput 0.10 Write jump 0.11 Write queue 0.08 Relative Fitness Bandwidth Throughput Predictor Score Predictor Score Read latency 1.00 Read latency 1.00 Write size 0.94 Write size 0.86 Write fraction 0.62 Write fraction 0.71 Read jump 0.45 Read jump 0.38 Bandwidth 0.44 Write latency 0.37 Latency 0.33 Read size 0.29 Read size 0.32 Latency 0.28 Write jump 0.30 Bandwidth 0.18 Write queue 0.23 Write queue 0.16
Latency Predictor Write queue Write fraction Read jump
Score 1.00 0.81 0.21
Latency Predictor Write queue Write jump Write fraction Write size Read size Read jump
Score 1.00 0.97 0.82 0.66 0.23 0.14
Latency Predictor Write queue Write size Write fraction Write jump Read latency Read size Read jump Bandwidth Throughput Latency
Score 1.00 0.87 0.69 0.67 0.45 0.42 0.35 0.34 0.22 0.21
Latency Predictor Latency Write fraction Write queue Bandwidth Read latency Write size Write latency Write jump Read jump
Score 1.00 0.80 0.28 0.28 0.26 0.20 0.13 0.13 0.09
Table I.4: The normalized importance measure of each predictor.
Model Absolute Relative Relative Performance Relative Fitness
Overall 2.6 2.1 2.5 2.4
Absolute Array Array Array Array Relative Array Array Array Array Relative Performance Array Array Array Array Relative Fitness Array Array Array Array
Absolute Array A Array B Array C Array D
A 4.0 -
Relative Array A Array B Array C Array D
A 4.0 4.0 2.0 3.0
Relative Performance Array A Array B Array C Array D
A 4.0 4.0 2.0 3.0
Relative Fitness Array A Array B Array C Array D
A 1 2.0 2.0 1.0
Bandwidth 2.2 2.2 2.8 2.2
Pairwise Array 3.0 Array 3.0 3.7 1.7 2.0 Array 4.0 4.0 1.7 2.3 Array 1 2.3 2.7 2.0
Bandwidth B C 2.0 2.0 Bandwidth B C 3.0 1.0 2.0 1.0 3.0 2.0 4.0 1.0 Bandwidth B C 4.0 1.0 2.0 1.0 3.0 5.0 4.0 5.0 Bandwidth B C 2.0 5.0 1 2.0 4.0 1 1.0 1.0
D 1.0 D 1.0 2.0 1.0 1.0 D 1.0 2.0 4.0 4.0 D 4.0 2.0 1.0 1
Array 3.3 Array 3.0 3.3 2.0 2.7 Array 3.7 3.3 2.0 2.7 Array 2.0 1 3.0 1.0
Throughput 2.5 2.1 2.6 2.7
Array 2.7 Array 1.0 2.3 2.7 1.0 Array 1.0 1.0 5.0 3.7 Array 3.3 1.7 1 1.0
Latency 3.0 1.9 2.2 2.3
Array 1.3 Array 1.3 3.0 1.0 1.3 Array 1.3 3.0 4.0 4.0 Array 3.7 3.3 3.0 1
Throughput B C D 4.0 2.0 1.0 Throughput A B C D 3.0 2.0 1.0 1.0 4.0 4.0 5.0 2.0 2.0 2.0 2.0 1.0 2.0 2.0 1.0 1.0 Throughput A B C D 4.0 4.0 1.0 1.0 4.0 4.0 1.0 3.0 2.0 2.0 5.0 4.0 2.0 2.0 5.0 4.0 Throughput A B C D 1 2.0 4.0 4.0 2.0 1 2.0 4.0 2.0 4.0 1 5.0 1.0 1.0 1.0 1 A 3.0 -
Table I.5: Tree sizes (leaf nodes) and their averages.
A 2.0 A 2.0 3.0 1.0 1.0 A 4.0 4.0 1.0 2.0 A 1 3.0 4.0 4.0
Latency B C 4.0 4.0 Latency B C 4.0 1.0 4.0 1.0 1.0 4.0 2.0 1.0 Latency B C 3.0 1.0 4.0 1.0 1.0 5.0 2.0 1.0 Latency B C 2.0 1.0 1 1.0 1.0 1 1.0 1.0
D 2.0 D 2.0 5.0 1.0 2.0 D 2.0 4.0 4.0 4.0 D 3.0 4.0 3.0 1
127
APPENDIX I. TPC-C MODEL TRAINING
Bandwidth error (TPC-C) Median relative error (Avg.)
Median relative error (Avg.)
Overall error (TPC-C) 0.15 0.14 0.13 0.12 0.11 0.1 0.09 0.08 0.07 0.06 0.05 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
0.12 0.11 0.1 0.09 0.08 0.07 0.06 0.05
70
30 AM (i=j) AM
RF
0.11 0.1 0.09 0.08 0.07 0.06 0.05 0.04 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
RM RM’
70 RF
Latency error (TPC-C) Median relative error (Avg.)
Median relative error (Avg.)
Throughput error (TPC-C)
40 50 60 Samples used for training (%)
70 RF
0.28 0.26 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
Figure I.7: Error vs. training. The training set size varies from 25% of the collected samples to 75%; the remaining samples are used for testing. For each training set size, the Median relative error is calculated for each pairing of devices. They are then averaged to produce the values shown.
128
Bandwidth error (TPC-C) Mean relative error (Avg.)
Mean relative error (Avg.)
Overall error (TPC-C) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
0.15 0.14 0.13 0.12 0.11 0.1 0.09 0.08
70
30
RF
AM (i=j) AM
0.13 0.12 0.11 0.1 0.09 0.08 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
RM RM’
70 RF
Latency error (TPC-C)
0.14
Mean relative error (Avg.)
Mean relative error (Avg.)
Throughput error (TPC-C)
40 50 60 Samples used for training (%)
70 RF
0.4 0.35 0.3 0.25 0.2 0.15 0.1 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
Figure I.8: Error vs. training. The training set size varies from 25% of the collected samples to 75%; the remaining samples are used for testing. For each training set size, the Mean relative error is calculated for each pairing of devices. They are then averaged to produce the values shown.
Appendix J
WorkloadMix model training Application Direct (raw) Buffered (sd) FS (fs) Postmark (phase 1) (pmc) Postmark (phase 2) (pmt) (srt) TPC-C (tpcc) Cache (cache) Total used
Samples 200 200 200 50 50 78 50 70
Iters 3 3 3 3 3 3 3 3
First sample 0 0 0 0 0 0 0 0 449
Last sample 99 99 99 24 24 38 24 34
Table J.1: Multiple training samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Array A 0.38 82.57 62.49 815.03 1856.84 24.44 3.82 23.29 440.48 26.97 45.03 11.47
Array B 0.38 83.25 62.46 818.80 1852.55 20.34 4.16 19.41 344.63 27.01 42.81 22.21
Array C 0.37 84.41 62.21 796.41 1894.78 20.33 3.68 14.83 626.66 63.01 106.23 25.84
Array D 0.38 82.85 62.35 828.24 1849.21 20.09 3.92 28.51 920.87 18.53 20.26 15.43
MAD 0.0 0.6 0.1 9.1 15.7 1.6 0.1 4.4 190.6 14.6 26.3 5.3
COV 0.9% 0.8% 0.2% 1.4% 1.0% 8.5% 4.5% 23.4% 37.7% 50.7% 59.5% 30.0%
Max. Diff. 2.7% 2.2% 0.5% 4.0% 2.5% 21.6% 13.0% 92.2% 167.2% 240.0% 424.3% 125.3%
Table J.2: Workload characteristics and performance are measured for each sample, on each storage device. The average value for each measurement is reported in this table. The mean absolute deviation (MAD), coefficient of variation (COV), and maximum relative differences of these averages are also reported; these metrics quantify how the averages change among the storage devices. For example, the mean average deviation of the averages for Read latency (ms) is 5.29, their coeefficient of variation is 29.994%, and the maximum relative difference is 125.28%.
129
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 1.1% 0.6% 0.1% 0.5% 0.8% 0.7% 1.5% 1.5% 0.5% 0.1% 0.3% 1.0%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 0.9% 0.7% 0.3% 0.2% 0.3% 0.8% 0.7% 1.9% 1.0% 1.8% 0.6% 1.9%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 0.9% 1.1% 0.4% 5.3% 0.2% 1.5% 1.1% 1.4% 0.1% 1.8% 2.4% 0.2%
Write fraction Write size (KB) Read size (KB) Write jump (MB) Read jump (MB) Write queue Read queue Bandwidth (MB/sec) Throughput (IO/sec) Latency (ms) Write latency (ms) Read latency (ms)
Rel. Diff. 0.9% 0.2% 0.0% 0.8% 0.0% 0.4% 0.6% 2.6% 2.1% 1.1% 0.2% 0.6%
Array A Min 25% 0.0 0.1 0.0 7.6 0.0 8.0 0.0 1.0 0.0 314.0 0.0 6.2 0.0 1.1 0.5 11.9 102.0 210.0 0.4 10.6 0.0 10.3 0.0 5.3 Array B Min 25% 0.0 0.1 0.0 7.6 0.0 8.0 0.0 0.0 0.0 318.0 0.0 1.1 0.0 1.1 0.3 9.0 65.0 165.0 0.8 12.7 0.0 2.4 0.0 9.0 Array C Min 25% 0.0 0.1 0.0 7.6 0.0 8.0 0.0 0.0 0.0 321.0 0.0 1.2 0.0 1.1 0.3 6.0 34.0 113.0 0.2 21.8 0.0 19.9 0.0 5.1 Array D Min 25% 0.0 0.1 0.0 7.3 0.0 8.0 0.0 1.0 0.0 324.0 0.0 1.1 0.0 1.1 0.5 12.5 92.0 200.0 0.1 7.6 0.0 1.0 0.0 6.9
50% 0.4 78.5 45.1 99.0 1159.0 19.6 1.8 20.6 277.0 20.3 43.3 7.6
75% 0.6 146.0 97.0 1259.0 3240.0 42.3 4.6 30.4 428.0 33.5 66.0 13.9
90% 0.8 188.8 141.0 3074.0 4759.0 57.9 10.4 45.3 844.0 55.4 93.8 22.7
Max 1.0 251.3 255.3 5525.0 6304.0 84.2 31.7 103.4 4253.0 138.6 165.7 89.7
Mean 0.4 82.6 62.5 815.0 1856.8 24.4 3.8 23.3 440.5 27.0 45.0 11.5
50% 0.4 80.7 44.7 94.0 1184.0 17.3 2.0 16.8 234.0 19.1 30.4 13.0
75% 0.6 149.4 97.6 1244.0 3223.0 33.2 5.1 26.0 337.0 27.9 59.0 21.6
90% 0.8 190.6 141.5 2927.0 4683.0 52.5 10.9 39.1 495.0 52.2 120.0 41.6
Max 1.0 251.0 255.1 5589.0 6115.0 64.0 24.3 104.7 9477.0 264.0 453.3 264.1
Mean 0.4 83.2 62.5 818.8 1852.6 20.3 4.2 19.4 344.6 27.0 42.8 22.2
50% 0.4 78.5 44.8 52.0 1241.0 15.3 1.7 11.3 153.0 44.5 108.0 11.6
75% 0.6 154.0 99.1 1087.0 3226.0 34.9 4.7 15.5 248.0 72.2 138.1 25.5
90% 0.8 193.8 139.9 3129.0 4863.0 53.0 9.9 22.2 516.0 129.4 220.9 60.5
Max 1.0 251.0 255.5 16383.0 6406.0 64.0 22.9 109.6 25519.0 578.0 583.5 423.9
Mean 0.4 84.4 62.2 796.4 1894.8 20.3 3.7 14.8 626.7 63.0 106.2 25.8
50% 0.4 79.1 45.2 103.0 1182.0 15.4 2.0 20.6 305.0 12.5 10.5 11.1
75% 0.6 148.0 97.6 1358.0 3188.0 29.7 5.0 38.4 530.0 18.8 26.5 15.8
90% 0.8 189.4 141.9 3028.0 4738.0 58.9 10.3 59.7 1097.0 34.0 42.5 31.9
Max 1.0 251.0 255.8 5519.0 6310.0 64.0 29.0 111.0 27676.0 211.9 214.8 139.7
Mean 0.4 82.9 62.3 828.2 1849.2 20.1 3.9 28.5 920.9 18.5 20.3 15.4
Table J.3: Workload characteristics and performance are measured for each sample, on each storage device. The minimum value, percentiles, maximum value and average are reported for each measurement. In addition, the relative difference between the average performance of the best and second-to-best iteration is reported. This value quantifies the change in a given measurement across multiple runs of the same sample on the same storage device.
131
APPENDIX J. WORKLOADMIX MODEL TRAINING 1
120
Array A Array B Array C Array D
0.9 100 Bandwidth (MB/sec)
0.8
Fraction
0.7 0.6 0.5 0.4 0.3
50
40
30
20
10
0
0
60 40 20 0 0
80
0.1
70
0.2
60
Array A Array B Array C Array D
80
50 100 150 200 250 300 350 400 450 Sample number
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
30000
Throughput (IO/sec)
15000 10000
1800
5000 1600
1400
1200
1000
800
600
400
200
Array A Array B Array C Array D
20000
0 0
Throughput (IO/sec) 600
0.8
500 Latency (ms)
1
0.6 0.4
Latency (ms)
50 100 150 200 250 300 350 400 450 Sample number
Array A Array B Array C Array D
400 300 200
200
100 180
120
100
80
60
40
0
20
0
160
Array A Array B Array C Array D
0.2
140
Fraction
Array A Array B Array C Array D
25000
0
Fraction
Bandwidth (MB/sec)
0 0
50 100 150 200 250 300 350 400 450 Sample number
Figure J.1: The cumulative distribution of performance is shown (top). In addition, the performance of each array is shown (bottom), sorted by the performance of Array C.
132
Throughput (IO/sec) on Array A 0.25
0.3 0.2
0.25 Fraction
0.2 0.15 0.1 0.05 0
<7
[7,
14
)
[ [ [ [ [ [ > 33 233, 364, 495, 626, 756, 887, =10 1 36 4 6 7 8 1 4) 95) 26) 56) 87) 018 8 )
[2 [2 [3 [4 > , 2 0, 2 7, 3 4, 4 0, 4 =47 0) 7) 4) 0) 7)
[11
0.3
0.35
0.25
0.3
0.2
0.25
0.15
43
[17 [22 [28 [33 >= 3 , , , , 7) 22) 28) 33) 39) 9
0.05
[14
[2 [2 [3 [4 [5 > 3, 21, 99, 78, 56, 34, =612 22 2 3 4 5 6 1) 99) 78) 56) 34) 12)
0
0.3
0.6
0.25 Fraction
0.5 0.4 0.3 0.2
0.1
0.1 0
0.05 0
<6
[6,
12
)
[12
[1 [2 [3 [3 > , 1 8, 2 4, 3 0, 3 6, 4 =42 8) 4) 0) 6) 2)
<1
[ [ [ [ [ [ > 78 178, 322, 466, 610, 753, 897, =10 4 32 4 6 7 8 1 2) 66) 10) 53) 97) 041 1 )
0
<1
0
[10
[21 [31 [41 [51 [61 >= 7 , , , , , 1) 31) 41) 51) 61) 71) 1
,2
0
<2
4
0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
[24
[49 [73 [97 [12 [14 >= 1 , , , 1 6 9) 73) 97) 121 , 14 , 17 70 ) 6) 0)
,4
Latency (ms) on Array D 0.3 0.25 Fraction
Fraction
0.1 0.05
0.1
Throughput (IO/sec) on Array D
0.25
0.15
0.2 0.15
0.05
Bandwidth (MB/sec) on Array D 0.3
0.2
[20 [30 [39 [49 [59 >= 6 , , , , , 0) 30) 39) 49) 59) 68) 8
,2
Latency (ms) on Array C 0.3
0.15
[10
Throughput (IO/sec) on Array C 0.7
0.2
<1
0
0.35
Fraction
Fraction
0.2 0.15 0.1
<1
,1
0.25
[19 [28 [37 [47 [56 >= 6 , , , , , 9) 28) 37) 47) 56) 65) 5
,1
Latency (ms) on Array B 0.4
Fraction
Fraction
Fraction
)
[10
Throughput (IO/sec) on Array B
0
11
<1
0
0.35
Bandwidth (MB/sec) on Array C
Fraction
0
0.05
[6,
0.1
<2
[14
0.1
<6
0.15
0.05
Bandwidth (MB/sec) on Array B 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
Latency (ms) on Array A
0.35
Fraction
Fraction
Bandwidth (MB/sec) on Array A 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<3
[ [ [ [ [ [ > 00 300, 507, 714, 922, 1130 1337 =15 4 , , 50 7 9 1 7) 14) 22) 130 133 154 4 ) 7) 4)
0.2 0.15 0.1 0.05 0
Figure J.2: Probability distributions of performance.
<6
[6,
12
)
[12
[18 [24 [29 [35 >= 4 , , , , 8) 24) 29) 35) 41) 1
,1
133
APPENDIX J. WORKLOADMIX MODEL TRAINING
Array A to Array B (min 0.15, max 8.00) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .29 0.29 0.43 0.57 0.71 0.85 0.99 =1.1 ,0 , , , , , .43 0.57 0.71 0.85 0.99 1.13 3 ) ) ) ) ) )
Array A to Array C (min 0.08, max 12.66) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array B to Array A (min 0.12, max 6.86) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .42 0.42 0.72 1.02 1.32 1.61 1.91 =2.2 ,0 , , , , , .72 1.02 1.32 1.61 1.91 2.21 1 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .53 0.53 0.99 1.44 1.89 2.34 2.80 =3.2 ,0 , , , , , .99 1.44 1.89 2.34 2.80 3.25 5 ) ) ) ) ) )
0.25 0.2 0.15 0.1 0.05
<0
[ [ [ [ [ [ > .26 0.26 0.44 0.61 0.79 0.97 1.15 =1.3 ,0 , , , , , .44 0.61 0.79 0.97 1.15 1.32 2 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .42 0.42 0.83 1.24 1.65 2.06 2.47 =2.8 ,0 , , , , , .83 1.24 1.65 2.06 2.47 2.88 8 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .56 0.56 1.05 1.54 2.02 2.51 3.00 =3.4 ,1 , , , , , .05 1.54 2.02 2.51 3.00 3.49 9 ) ) ) ) ) )
Array D to Array A (min 0.07, max 2.53)
0.2
0.25
0
.23
<1
[ [ [ [ [ [ > .16 1.16 1.66 2.17 2.67 3.17 3.67 =4.1 ,1 , , , , , .66 2.17 2.67 3.17 3.67 4.17 7 ) ) ) ) ) )
0
Array D to Array C (min 0.01, max 1.51)
0.05
0.05
[ [ [ [ [ > 23 0.39 0.54 0.70 0.85 1.01 =1.1 ,0 , , , , , .39 0.54 0.70 0.85 1.01 1.17 7 ) ) ) ) ) )
Array C to Array D (min 0.66, max 92.28) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.1
0.1
[0.
[ [ [ [ [ [ > .98 0.98 1.32 1.66 2.00 2.34 2.68 =3.0 ,1 , , , , , .32 1.66 2.00 2.34 2.68 3.02 2 ) ) ) ) ) )
0.15
0.2
<0
<0
0.2
0.15 0.05
[ [ [ [ [ [ > .64 0.64 0.88 1.12 1.36 1.60 1.84 =2.0 ,0 , , , , , .88 1.12 1.36 1.60 1.84 2.08 8 ) ) ) ) ) )
Array D to Array B (min 0.04, max 1.56) 0.3
0.1
<0
Array B to Array D (min 0.64, max 23.19) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array B (min 0.07, max 95.72) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.25
0.15
0
Array B to Array C (min 0.01, max 14.06) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array A (min 0.08, max 11.97) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array A to Array D (min 0.39, max 13.60) 0.3
<0
[ [ [ [ [ [ > .17 0.17 0.30 0.42 0.55 0.67 0.80 =0.9 ,0 , , , , , .30 0.42 0.55 0.67 0.80 0.92 2 ) ) ) ) ) )
0
<0
[ [ [ [ [ [ > .12 0.12 0.24 0.35 0.47 0.58 0.69 =0.8 ,0 , , , , , .24 0.35 0.47 0.58 0.69 0.81 1 ) ) ) ) ) )
Figure J.3: Each graph shows the probability distribution of the Bandwidth relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
134
Array A to Array B (min 0.15, max 8.00) 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .29 0.29 0.43 0.57 0.70 0.84 0.98 =1.1 ,0 , , , , , .43 0.57 0.70 0.84 0.98 1.12 2 ) ) ) ) ) )
Array A to Array C (min 0.08, max 12.49) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array B to Array A (min 0.12, max 6.72) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .38 0.38 0.64 0.89 1.15 1.40 1.66 =1.9 ,0 , , , , , .64 0.89 1.15 1.40 1.66 1.91 1 ) ) ) ) ) )
[ [ [ [ [ [ > .26 0.26 0.44 0.63 0.81 0.99 1.17 =1.3 ,0 , , , , , .44 0.63 0.81 0.99 1.17 1.35 5 ) ) ) ) ) )
Array B to Array C (min 0.01, max 14.83) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array A (min 0.08, max 12.08) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
<0
[ [ [ [ [ [ > .36 0.36 0.70 1.04 1.39 1.73 2.08 =2.4 ,0 , , , , , .70 1.04 1.39 1.73 2.08 2.42 2 ) ) ) ) ) )
0.25 0.2 0.15 0.1 0.05
[ [ [ [ [ [ > .53 0.53 0.98 1.43 1.88 2.33 2.78 =3.2 ,0 , , , , , .98 1.43 1.88 2.33 2.78 3.23 3 ) ) ) ) ) )
0
<0
[ [ [ [ [ [ > .55 0.55 1.03 1.52 2.00 2.48 2.97 =3.4 ,1 , , , , , .03 1.52 2.00 2.48 2.97 3.45 5 ) ) ) ) ) )
Array D to Array A (min 0.07, max 2.55) 0.3
0.2
0.25 0.2 0.15
0.1
0.1
0.05 0
0.05
<0
[ [ [ [ [ [ > .23 0.23 0.39 0.54 0.70 0.86 1.02 =1.1 ,0 , , , , , .39 0.54 0.70 0.86 1.02 1.17 7 ) ) ) ) ) )
[ [ [ [ [ [ > .67 0.67 0.95 1.23 1.50 1.78 2.06 =2.3 ,0 , , , , , .95 1.23 1.50 1.78 2.06 2.34 4 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .98 0.98 1.31 1.65 1.98 2.32 2.65 =2.9 ,1 , , , , , .31 1.65 1.98 2.32 2.65 2.99 9 ) ) ) ) ) )
Array C to Array D (min 0.66, max 93.19) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<1
[ [ [ [ [ [ > .19 1.19 1.72 2.25 2.77 3.30 3.83 =4.3 ,1 , , , , , .72 2.25 2.77 3.30 3.83 4.36 6 ) ) ) ) ) )
Array D to Array B (min 0.04, max 1.56)
0.25
0.15
<0
Array B to Array D (min 0.64, max 23.24) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array C to Array B (min 0.07, max 96.70) 0.3
<0
Array A to Array D (min 0.39, max 13.67) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0
<0
[ [ [ [ [ [ > .17 0.17 0.30 0.42 0.55 0.68 0.80 =0.9 ,0 , , , , , .30 0.42 0.55 0.68 0.80 0.93 3 ) ) ) ) ) )
Array D to Array C (min 0.01, max 1.51) 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0
<0
[ [ [ [ [ [ > .12 0.12 0.24 0.35 0.46 0.58 0.69 =0.8 ,0 , , , , , .24 0.35 0.46 0.58 0.69 0.80 0 ) ) ) ) ) )
Figure J.4: Each graph shows the probability distribution of the Throughput relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
135
APPENDIX J. WORKLOADMIX MODEL TRAINING
Array A to Array B (min 0.12, max 6.57) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array A to Array C (min 0.03, max 12.20) 0.3
0.2
0.25 0.2
0.15
0.15 0.1
0.1
0.05 0
<0
[ [ [ [ [ [ > .52 0.52 0.92 1.33 1.73 2.13 2.54 =2.9 ,0 , , , , , .92 1.33 1.73 2.13 2.54 2.94 4 ) ) ) ) ) )
0.05
<0
[ [ [ [ [ [ > .51 0.51 0.98 1.45 1.92 2.39 2.87 =3.3 ,0 , , , , , .98 1.45 1.92 2.39 2.87 3.34 4 ) ) ) ) ) )
Array B to Array A (min 0.15, max 8.59) 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<0
[ [ [ [ [ [ > .40 0.40 0.65 0.90 1.15 1.40 1.65 =1.8 ,0 , , , , , .65 0.90 1.15 1.40 1.65 1.89 9 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .28 0.28 0.48 0.68 0.88 1.08 1.28 =1.4 ,0 , , , , , .48 0.68 0.88 1.08 1.28 1.48 8 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .67 0.67 1.29 1.90 2.52 3.14 3.76 =4.3 ,1 , , , , , .29 1.90 2.52 3.14 3.76 4.38 8 ) ) ) ) ) )
[ [ [ [ [ [ > .93 0.93 1.36 1.79 2.22 2.65 3.08 =3.5 ,1 , , , , , .36 1.79 2.22 2.65 3.08 3.51 1 ) ) ) ) ) )
[ [ [ [ [ [ > .17 0.17 0.31 0.46 0.60 0.75 0.89 =1.0 ,0 , , , , , .31 0.46 0.60 0.75 0.89 1.04 4 ) ) ) ) ) )
Array B to Array D (min 0.04, max 1.65) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
[ [ [ [ [ [ > .16 0.16 0.28 0.41 0.53 0.65 0.78 =0.9 ,0 , , , , , .28 0.41 0.53 0.65 0.78 0.90 0 ) ) ) ) ) )
Array C to Array D (min 0.01, max 1.04) 0.3 0.25 0.2 0.15 0.1 0.05
<0
[ [ [ [ [ [ > .54 0.54 1.08 1.61 2.15 2.68 3.22 =3.7 ,1 , , , , , .08 1.61 2.15 2.68 3.22 3.75 5 ) ) ) ) ) )
0
Array D to Array B (min 0.61, max 28.56) 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
<0
<0
Array C to Array B (min 0.01, max 20.56) 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Array D to Array A (min 0.50, max 40.28) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0
Array B to Array C (min 0.05, max 104.81) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
Array C to Array A (min 0.08, max 29.00) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Array A to Array D (min 0.02, max 2.00)
0.25
<0
[ [ [ [ [ [ > .11 0.11 0.21 0.31 0.42 0.52 0.62 =0.7 ,0 , , , , , .21 0.31 0.42 0.52 0.62 0.72 2 ) ) ) ) ) )
Array D to Array C (min 0.96, max 111.99) 0.3 0.25 0.2 0.15 0.1 0.05
<1
[ [ [ [ [ [ > .97 1.97 3.33 4.70 6.06 7.42 8.78 =10 ,3 , , , , , . .33 4.70 6.06 7.42 8.78 10.1 15 ) ) ) ) ) 5)
0
<1
[ [ [ [ [ [ > .75 1.75 2.54 3.33 4.12 4.91 5.70 =6.4 ,2 , , , , , .54 3.33 4.12 4.91 5.70 6.49 9 ) ) ) ) ) )
Figure J.5: Each graph shows the probability distribution of the Latency relative fitness values for a given device pairing, over all workload samples. A relative fitness value is the ratio of the performance of two devices. Values greater than 1.0 indicate a performance increase as a workload is moved from device A to device B, values less than zero indicate a decrease, and values close to zero indicate little change.
1
1
0.9
0.9
0.8
0.8 0.7 Fraction
0.7 0.6 0.5
0.6 0.5 0.4
Write fraction
250
150
100
0
0.1
200
0.2 1
0.6
0.5
0.4
0.3
0.2
0.1
0
0.2
Array A Array B Array C Array D
0.3
0.9
0.3
0.8
Array A Array B Array C Array D
50
0.4
0.7
Fraction
136
Write size (KB)
1
1
0.9
0.9
0.8 0.8 Fraction
0.6 0.5
0.7 0.6
Write queue
Write jump (MB)
Figure J.6: The cumulative distribution of workload characteristics.
4500
4000
3500
2500
2000
1500
1000
500
0
0.4 70
40
30
20
10
0.1
Array A Array B Array C Array D
0.5
60
0.2
50
Array A Array B Array C Array D
0.3
3000
0.4
0
Fraction
0.7
137
APPENDIX J. WORKLOADMIX MODEL TRAINING
Overall Predictor Read queue Read size Write fraction Write size Read jump Write queue Write jump
Score 1.00 0.88 0.79 0.64 0.62 0.59 0.31
Overall Predictor Read queue Read size Write size Read jump Write fraction Write queue Write jump
Score 1.00 0.93 0.69 0.67 0.67 0.50 0.26
Overall Predictor Bandwidth Latency Throughput Write fraction Read queue Read jump Read latency Write queue Read size Write size Write jump Write latency
Score 1.00 0.94 0.93 0.36 0.33 0.33 0.30 0.26 0.26 0.22 0.17 0.12
Overall Predictor Bandwidth Latency Throughput Read jump Read latency Write fraction Read size Read queue Write queue Write latency Write size Write jump
Score 1.00 1.00 0.99 0.94 0.89 0.89 0.53 0.51 0.50 0.45 0.40 0.33
Absolute Bandwidth Throughput Predictor Score Predictor Score Read size 1.00 Read queue 1.00 Read queue 0.87 Read size 0.90 Write size 0.72 Read jump 0.63 Write fraction 0.67 Write size 0.55 Read jump 0.53 Write fraction 0.50 Write jump 0.36 Write queue 0.44 Write queue 0.27 Write jump 0.40 Relative Bandwidth Throughput Predictor Score Predictor Score Read size 1.00 Read queue 1.00 Read queue 0.82 Read size 0.84 Write size 0.75 Read jump 0.74 Write fraction 0.60 Write size 0.56 Read jump 0.58 Write queue 0.41 Write jump 0.32 Write fraction 0.41 Write queue 0.18 Write jump 0.33 Relative Performance Bandwidth Throughput Predictor Score Predictor Score Bandwidth 1.00 Throughput 1.00 Latency 0.21 Read queue 0.25 Write fraction 0.17 Bandwidth 0.17 Read jump 0.17 Write fraction 0.16 Read latency 0.16 Read jump 0.14 Throughput 0.14 Read size 0.13 Read size 0.11 Write queue 0.12 Write jump 0.11 Write size 0.11 Write size 0.09 Latency 0.11 Read queue 0.08 Write jump 0.07 Write queue 0.08 Read latency 0.05 Write latency 0.05 Write latency 0.04 Relative Fitness Bandwidth Throughput Predictor Score Predictor Score Read jump 1.00 Read latency 1.00 Read latency 0.97 Bandwidth 0.96 Bandwidth 0.96 Throughput 0.94 Throughput 0.95 Read jump 0.83 Latency 0.93 Write fraction 0.78 Write fraction 0.75 Latency 0.65 Read queue 0.57 Write queue 0.47 Read size 0.50 Read queue 0.46 Write latency 0.44 Read size 0.41 Write size 0.36 Write latency 0.36 Write queue 0.31 Write size 0.29 Write jump 0.24 Write jump 0.26
Latency Predictor Write fraction Read queue Write queue Read jump Read size Write size Write jump
Score 1.00 0.90 0.89 0.57 0.55 0.49 0.11
Latency Predictor Read queue Write fraction Write queue Read size Write size Read jump Write jump
Score 1.00 0.85 0.84 0.74 0.60 0.57 0.09
Latency Predictor Latency Bandwidth Read latency Write fraction Write queue Read jump Read queue Read size Write size Throughput Write latency Write jump
Score 1.00 0.21 0.21 0.16 0.16 0.13 0.11 0.11 0.10 0.08 0.08 0.05
Latency Predictor Latency Write fraction Bandwidth Throughput Read jump Write queue Read size Read latency Write size Write latency Write jump Read queue
Score 1.00 0.78 0.71 0.71 0.64 0.49 0.46 0.42 0.39 0.37 0.34 0.33
Table J.4: The normalized importance measure of each predictor.
Model Absolute Relative Relative Performance Relative Fitness
Overall 74.5 68.8 67.7 57.9
Absolute Array Array Array Array Relative Array Array Array Array Relative Performance Array Array Array Array Relative Fitness Array Array Array Array
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Pairwise Array 68.7 Array 68.7 83.3 67.7 74.0 Array 74.0 86.0 52.7 92.7 Array 1 30.3 81.7 43.7
Bandwidth B C 76.0 66.0 Bandwidth A B C 83.0 80.0 96.0 89.0 76.0 79.0 101.0 91.0 66.0 88.0 84.0 95.0 Bandwidth A B C 74.0 62.0 17.0 91.0 75.0 85.0 47.0 94.0 73.0 86.0 66.0 101.0 Bandwidth A B C 1 92.0 98.0 32.0 1 34.0 85.0 43.0 1 55.0 56.0 46.0 A 83.0 -
Bandwidth 78.8 85.6 71.9 58.2
Array 86.0 Array 77.3 86.0 61.3 76.0 Array 66.7 75.0 83.0 63.0 Array 64.0 1 78.7 56.3
D 90.0
A 64.0 -
D 88.0 53.0 83.0 90.0
A 64.0 88.0 42.0 67.0
D 98.0 56.0 60.0 70.0
A 76.0 82.0 65.0 91.0
D 25.0 73.0 60.0 1
A 1 39.0 74.0 36.0
Throughput 69.2 55.2 64.3 52.6
Array 61.7 Array 68.7 34.7 61.7 57.3 Array 31.3 63.0 72.0 74.3 Array 65.7 52.3 1 34.3
Throughput B C 88.0 43.0 Throughput B C 84.0 60.0 88.0 5.0 9.0 43.0 73.0 32.0 Throughput B C 107.0 2.0 76.0 64.0 66.0 73.0 83.0 84.0 Throughput B C 62.0 54.0 1 34.0 95.0 1 76.0 23.0
Latency 75.5 65.5 66.8 62.8
Array 81.7 Array 76.3 71.7 77.0 81.7 Array 80.0 43.0 76.7 72.3 Array 48.3 67.0 72.0 1
D 82.0
A 59.0 -
D 63.0 78.0 62.0 82.0
A 59.0 73.0 60.0 67.0
D 41.0 22.0 65.0 74.0
A 72.0 85.0 46.0 101.0
D 54.0 26.0 58.0 1
A 1 20.0 86.0 40.0
Table J.5: Tree sizes (leaf nodes) and their averages.
Latency B C 94.0 76.0 Latency B C 68.0 50.0 94.0 20.0 84.0 76.0 71.0 45.0 Latency B C 31.0 75.0 74.0 40.0 89.0 70.0 40.0 38.0 Latency B C 38.0 45.0 1 89.0 98.0 1 37.0 34.0
D 73.0 D 78.0 84.0 86.0 73.0 D 101.0 51.0 105.0 73.0 D 66.0 102.0 98.0 1
139
APPENDIX J. WORKLOADMIX MODEL TRAINING
Bandwidth error (WorkloadMix) Median relative error (Avg.)
Median relative error (Avg.)
Overall error (WorkloadMix) 0.26 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08
70
30 AM (i=j) AM
RF
0.26 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
RM RM’
70 RF
Latency error (WorkloadMix) Median relative error (Avg.)
Median relative error (Avg.)
Throughput error (WorkloadMix)
40 50 60 Samples used for training (%)
70 RF
0.28 0.26 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
Figure J.7: Error vs. training. The training set size varies from 25% of the collected samples to 75%; the remaining samples are used for testing. For each training set size, the Median relative error is calculated for each pairing of devices. They are then averaged to produce the values shown.
140
Bandwidth error (WorkloadMix) Mean relative error (Avg.)
Mean relative error (Avg.)
Overall error (WorkloadMix) 0.75 0.7 0.65 0.6 0.55 0.5 0.45 0.4 0.35 0.3 0.25 0.2 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
0.7 0.65 0.6 0.55 0.5 0.45 0.4 0.35 0.3 0.25 0.2
70
30
RF
AM (i=j) AM
0.5 0.45 0.4 0.35 0.3 0.25 0.2 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
RM RM’
70 RF
Latency error (WorkloadMix)
0.55
Mean relative error (Avg.)
Mean relative error (Avg.)
Throughput error (WorkloadMix)
40 50 60 Samples used for training (%)
70 RF
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 30 AM (i=j) AM
40 50 60 Samples used for training (%) RM RM’
70 RF
Figure J.8: Error vs. training. The training set size varies from 25% of the collected samples to 75%; the remaining samples are used for testing. For each training set size, the Mean relative error is calculated for each pairing of devices. They are then averaged to produce the values shown.
Appendix K
FitnessDirect model testing on FitnessDirect samples Application Direct (raw) Total used
Samples 200
Iters 3
First sample 100 100
Last sample 199
Table K.1: Multiple testing samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
141
142
Overall (FitnessDirect)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Overall (FitnessDirect)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Overall (FitnessDirect) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Overall (FitnessDirect)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Overall (FitnessDirect)
Relative error
Figure K.1: The cumulative distribution of relative error over all pairwise predictions.
APPENDIX K. FITNESSDIRECT MODEL TESTING ON FITNESSDIRECT SAMPLES
Bandwidth (FitnessDirect)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error Bandwidth (FitnessDirect)
Bandwidth (FitnessDirect)
1 1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Relative Performance Relative Fitness 0
Fraction 1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative error
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.9
Relative error Bandwidth (FitnessDirect)
0.9
0
1
0.9
0.8
0.7
0.6
0.4
0.2
0
0.3
Relative error
Relative Relative Performance 0
Absolute Relative
Bandwidth (FitnessDirect) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.1
Fraction
0.1
0
0.5
Absolute (i=j) Absolute
0.1
0.7
0.2
0.6
0.3
0.5
0.4
0.4
0.5
0.3
0.6
Fraction
Fraction
0.7
0.2
0.8
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.1
0.9
Relative error
Figure K.2: The cumulative distribution of relative error over all pairwise predictions.
143
144
Throughput (FitnessDirect)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Performance Relative Fitness 0.1
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Throughput (FitnessDirect)
0.9
Relative error Throughput (FitnessDirect)
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute Relative 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Throughput (FitnessDirect)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Throughput (FitnessDirect)
Relative error
Figure K.3: The cumulative distribution of relative error over all pairwise predictions.
APPENDIX K. FITNESSDIRECT MODEL TESTING ON FITNESSDIRECT SAMPLES
Latency (FitnessDirect)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Latency (FitnessDirect)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Latency (FitnessDirect) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Latency (FitnessDirect)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Latency (FitnessDirect)
Relative error
Figure K.4: The cumulative distribution of relative error over all pairwise predictions.
145
146
<-0
[[0 [1 [1 [2 [2 > .06 0.06 .48, .02, .57, .11, .65, =3.2 ,0 0 .48 1.02 1.57 2.11 2.65 3.20 ) ) ) ) ) )
0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[0 [0 [1 [1 [2 > .14 0.14 .29, .72, .15, .58, .01, =2.4 ,0 4 .29 0.72 1.15 1.58 2.01 2.44 ) ) ) ) ) )
Fraction
0.1 0.05 0
<-0
[[[0 [0 [0 [0 > .30 0.30 0.12 .07, .25, .43, .61, =0.7 , -0 , 0 9 0 0 0 0 .12 .07 .25) .43) .61) .79) ) )
<-0
[[[0 [0 [0 [0 > .38 0.38 0.15 .08, .31, .54, .77, =1.0 , -0 , 0 0 0 0 0 1 .15 .08 .31) .54) .77) .00) ) )
[[[0 [0 [0 [0 > .33 0.33 0.09 .15, .39, .63, .87, =1.1 , -0 , 0 1 0 0 0 1 .09 .15 .39) .63) .87) .11) ) )
Fraction
<-0
[[[0 [0 [0 [0 > .32 0.32 0.09 .15, .38, .61, .85, =1.0 , -0 , 0 8 0 0 0 1 .09 .15 .38) .61) .85) .08) ) )
Fraction <-0
[[[0 [0 [0 [0 > .29 0.29 0.12 .06, .24, .41, .59, =0.7 , -0 , 0 7 0 0 0 0 .12 .06 .24) .41) .59) .77) ) )
0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
<0
[ [ [ [ [ [ > .22 0.22 1.10 1.98 2.86 3.74 4.62 =5.5 ,1 , , , , , .10 1.98 2.86 3.74 4.62 5.50 0 ) ) ) ) ) )
Relative Performance: Latency (Avg )
Fraction [[0 [0 [0 [1 [1 > .27 0.27 .10, .47, .84, .20, .57, =1.9 ,0 4 .10 0.47 0.84 1.20 1.57 1.94 ) ) ) ) ) )
0.3 0.25 0.2 0.15 0.1 0.05 0
Absolute: Latency (Avg )
0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Fitness: Latency (Avg )
Fraction
Fraction
Fraction <-0
<-0
[[0 [0 [0 [0 [0 > .23 0.23 .02, .26, .50, .75, .99, =1.2 ,0 3 .02 0.26 0.50 0.75 0.99 1.23 ) ) ) ) ) )
Relative Fitness: Throughput (Avg ) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative: Latency (Avg ) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
Relative: Throughput (Avg )
0.2 0.15
Relative Performance: Throughput (Avg ) 0.3 0.25 0.2 0.15 0.1 0.05 0
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Absolute: Throughput (Avg ) 0.25 Fraction
Fraction
Relative Fitness: Bandwidth (Avg ) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Performance: Bandwidth (Avg )
Fraction
Relative: Bandwidth (Avg )
Fraction
Fraction
Absolute: Bandwidth (Avg ) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[0 [0 [0 [0 > .31 0.31 0.09 .14, .37, .59, .82, =1.0 , -0 , 0 4 0 0 0 1 .09 .14 .37) .59) .82) .04) ) )
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[0 [0 [0 [0 > .31 0.31 0.12 .06, .25, .43, .62, =0.8 , -0 , 0 0 0 0 0 0 .12 .06 .25) .43) .62) .80) ) )
Figure K.5: The probability distribution of relative error over all pairwise predictions.
APPENDIX K. FITNESSDIRECT MODEL TESTING ON FITNESSDIRECT SAMPLES
Bandwidth
Fraction
Fraction
Overall 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
0.3 0.25 0.2 0.15 0.1 0.05 0
, -0
80
)
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
, -0
80
)
Latency
Fraction
Fraction
Throughput 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
147
0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
Figure K.6: Probability distributions of the difference in the absolute value of the relative error value−measured value (| predictedmeasured |) between the absolute and relative fitness models, over all predictions. The x value axis represent the amount by which relative error is reduced. Negative values indicate an increase in error.
148
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 37 / 15 / 46 2 / 4 / 92 10 / 11 / 77 29 / 14 / 55 12 / 2 / 84 8 / 0 / 90 19 / 15 / 64 18 / 9 / 71 7 / 9 / 82 24 / 16 / 58 6 / 4 / 88 6 / 2 / 90 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 14 / 3 / 81 3 / 4 / 91 0 / 1 / 97 22 / 9 / 67 9 / 5 / 84 0 / 0 / 98 18 / 5 / 75 12 / 2 / 84 0 / 0 / 98 19 / 8 / 71 2 / 3 / 93 8 / 4 / 86 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 36 / 3 / 59 51 / 12 / 35 47 / 1 / 50 40 / 13 / 45 19 / 14 / 65 5 / 6 / 87 41 / 7 / 50 13 / 7 / 78 14 / 6 / 78 39 / 17 / 42 6 / 2 / 90 17 / 13 / 68 nan% nan% nan%
Bandwidth AM vs. RM RM vs. RM’ 32 / 48 / 18 29 / 66 / 3 39 / 19 / 40 30 / 62 / 6 29 / 32 / 37 44 / 48 / 6 34 / 60 / 4 25 / 56 / 17 55 / 43 / 0 42 / 55 / 1 36 / 28 / 34 31 / 60 / 7 40 / 44 / 14 42 / 53 / 3 28 / 34 / 36 51 / 38 / 9 33 / 32 / 33 48 / 43 / 7 29 / 54 / 15 37 / 52 / 9 39 / 37 / 22 27 / 54 / 17 58 / 40 / 0 28 / 63 / 7 100% 0% 0% 100% 0% 0% Throughput AM vs. RM RM vs. RM’ 50 / 41 / 7 59 / 39 / 0 34 / 51 / 13 61 / 35 / 2 50 / 48 / 0 0 / 0 / 98 36 / 60 / 2 54 / 39 / 5 37 / 48 / 13 61 / 37 / 0 49 / 49 / 0 24 / 74 / 0 34 / 60 / 4 61 / 37 / 0 26 / 63 / 9 55 / 40 / 3 49 / 49 / 0 46 / 51 / 1 44 / 52 / 2 33 / 60 / 5 39 / 52 / 7 41 / 51 / 6 38 / 44 / 16 68 / 29 / 1 100% 0% 0% 100% 0% 0% Latency AM vs. RM RM vs. RM’ 24 / 71 / 3 36 / 55 / 7 33 / 62 / 3 41 / 56 / 1 20 / 64 / 14 24 / 61 / 13 30 / 61 / 7 48 / 44 / 6 22 / 35 / 41 46 / 44 / 8 50 / 47 / 1 36 / 61 / 1 19 / 49 / 30 31 / 45 / 22 33 / 63 / 2 45 / 49 / 4 31 / 36 / 31 46 / 44 / 8 27 / 62 / 9 42 / 42 / 14 39 / 45 / 14 30 / 54 / 14 47 / 42 / 9 38 / 45 / 15 100% 0% 0% 100% 0% 0%
RM’ vs. RF 49 / 47 / 2 44 / 49 / 5 36 / 58 / 4 35 / 55 / 8 37 / 60 / 1 29 / 63 / 6 31 / 63 / 4 40 / 54 / 4 43 / 53 / 2 29 / 62 / 7 24 / 68 / 6 43 / 51 / 4 100% 0% 0%
AM vs. RF 28 / 68 / 2 42 / 56 / 0 38 / 57 / 3 19 / 74 / 5 33 / 63 / 2 28 / 66 / 4 29 / 67 / 2 40 / 51 / 7 43 / 50 / 5 22 / 73 / 3 32 / 64 / 2 35 / 58 / 5 100% 0% 0%
RM’ vs. RF 28 / 68 / 2 30 / 65 / 3 25 / 66 / 7 25 / 68 / 5 30 / 64 / 4 31 / 60 / 7 29 / 67 / 2 42 / 51 / 5 33 / 61 / 4 36 / 54 / 8 29 / 61 / 8 34 / 62 / 2 100% 0% 0%
AM vs. RF 33 / 62 / 3 40 / 55 / 3 25 / 72 / 1 35 / 61 / 2 37 / 59 / 2 18 / 76 / 4 29 / 68 / 1 35 / 58 / 5 37 / 59 / 2 25 / 72 / 1 29 / 65 / 4 38 / 56 / 4 100% 0% 0%
RM’ vs. RF 38 / 55 / 5 40 / 53 / 5 40 / 52 / 6 36 / 57 / 5 37 / 59 / 2 29 / 64 / 5 40 / 51 / 7 41 / 54 / 3 39 / 57 / 2 38 / 54 / 6 26 / 69 / 3 35 / 60 / 3 100% 0% 0%
AM vs. RF 14 / 81 / 3 22 / 75 / 1 18 / 80 / 0 25 / 69 / 4 28 / 68 / 2 18 / 77 / 3 22 / 70 / 6 31 / 66 / 1 31 / 64 / 3 20 / 77 / 1 21 / 73 / 4 36 / 59 / 3 100% 0% 0%
Table K.2: Models are compared across all pairwise predictions. For each prediction, a score of 1 is assigned to the model with the lowest accuracy. If the accuracy of two models is within 0.01, a score of 1 goes to the tie category (3rd number shown). Hypotheses are tested in turn. Specifically, idealized absolute models (i = j) are compared against in-practice absolute models (i 6= j), relative models are compared to absolute models (AM vs. RM), relative models that use performance are compared to those that do not (RM vs. RM’), and relative fitness models are compared to relative models that use performance (RM’ vs. RF). In addition, relative fitness models are compared directly to absolute models (AM vs. RF).
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.23 0.28 0.23 0.20 0.14
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.26 0.34 0.32 0.32
Relative Array A Array B Array C Array D
A 0.26 0.23 0.26 0.19
Relative Performance Array A Array B Array C Array D
A 0.04 0.13 0.19 0.16
Relative Fitness Array A Array B Array C Array D
A 0.0 0.12 0.12 0.12
Bandwidth 0.21 0.24 0.23 0.19 0.14
Pairwise Array A Array 0.20 0.30 0.28 0.22 0.30 0.25 0.28 0.22 Array A Array 0.20 0.24 0.20 0.22 0.20 0.18 0.19 0.20 Array A Array 0.04 0.19 0.17 0.04 0.19 0.20 0.16 0.16 Array A Array 0.0 0.14 0.11 0.0 0.13 0.18 0.12 0.09
Bandwidth B C 0.27 0.23 0.17 0.27 0.19 0.23 0.17 0.23 Bandwidth B C 0.23 0.24 0.17 0.30 0.17 0.23 0.17 0.30 Bandwidth B C 0.14 0.18 0.04 0.27 0.21 0.04 0.16 0.19 Bandwidth B C 0.15 0.18 0.0 0.20 0.18 0.0 0.09 0.20
D 0.18 0.23 0.18 0.22
A 0.20 0.23 0.27 0.24
D 0.18 0.22 0.19 0.22
A 0.20 0.17 0.18 0.21
D 0.16 0.19 0.19 0.03
A 0.02 0.19 0.24 0.15
D 0.12 0.12 0.18 0.0
A 0.0 0.11 0.14 0.11
B
B
B
B
Throughput 0.23 0.26 0.22 0.22 0.14
Array 0.31 0.30 0.24 0.28 Array 0.24 0.28 0.24 0.28 Array 0.21 0.28 0.04 0.28 Array 0.18 0.20 0.0 0.20
Throughput B C 0.26 0.23 0.22 0.24 0.26 0.23 0.22 0.24 Throughput B C 0.28 0.19 0.22 0.22 0.16 0.23 0.19 0.21 Throughput B C 0.27 0.27 0.03 0.34 0.19 0.03 0.15 0.34 Throughput B C 0.16 0.16 0.0 0.20 0.18 0.0 0.07 0.19
Table K.3: Median relative error
C
C
C
C
Latency 0.25 0.34 0.25 0.20 0.14
Array 0.31 0.27 0.26 0.26 Array 0.25 0.29 0.25 0.26 Array 0.21 0.18 0.23 0.04 Array 0.13 0.12 0.19 0.0
D 0.29 0.29 0.29 0.29
A 0.18 0.27 0.30 0.27
D 0.32 0.32 0.32 0.29
A 0.18 0.16 0.18 0.18
D 0.32 0.13 0.28 0.03
A 0.05 0.18 0.14 0.18
D 0.12 0.08 0.18 0.0
A 0.0 0.10 0.12 0.12
D
D
D
D
Latency B C 0.44 0.58 0.26 0.33 0.32 0.28 0.31 0.31 Latency B C 0.21 0.31 0.26 0.32 0.20 0.28 0.24 0.31 Latency B C 0.16 0.22 0.05 0.26 0.20 0.07 0.17 0.29 Latency B C 0.12 0.19 0.0 0.20 0.18 0.0 0.10 0.18
D 0.77 0.33 0.33 0.34 D 0.29 0.31 0.25 0.34 D 0.16 0.22 0.23 0.07 D 0.15 0.14 0.19 0.0
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.47 0.71 0.46 0.35 0.24
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.66 1.02 0.98 0.97
Relative Array A Array B Array C Array D
A 0.66 0.58 0.70 0.53
Relative Performance Array A Array B Array C Array D
A 0.09 0.38 0.46 0.33
Relative Fitness Array A Array B Array C Array D
A 0.0 0.25 0.21 0.14
Bandwidth 0.62 0.72 0.56 0.36 0.24
Pairwise Array A Array 0.38 1.23 0.54 0.45 0.53 0.49 0.52 0.45 Array A Array 0.38 0.39 0.41 0.45 0.44 0.37 0.39 0.40 Array A Array 0.08 0.32 0.31 0.08 0.34 0.38 0.26 0.22 Array A Array 0.0 0.22 0.23 0.0 0.20 0.31 0.15 0.13
Bandwidth B C 0.59 0.72 0.40 1.34 0.41 0.99 0.36 1.03 Bandwidth B C 0.45 0.75 0.40 0.82 0.40 0.99 0.41 0.82 Bandwidth B C 0.25 0.50 0.08 0.62 0.37 0.13 0.23 0.35 Bandwidth B C 0.23 0.23 0.0 0.32 0.34 0.0 0.14 0.31
D 0.37 0.49 0.38 0.41
A 0.24 0.28 0.29 0.27
D 0.40 0.52 0.39 0.41
A 0.24 0.30 0.32 0.35
D 0.32 0.23 0.27 0.05
A 0.04 0.32 0.31 0.20
D 0.20 0.18 0.29 0.0
A 0.0 0.26 0.21 0.14
B
B
B
B
Throughput 0.35 0.37 0.35 0.35 0.24
Array 1.02 0.70 0.57 0.59 Array 0.48 0.61 0.57 0.53 Array 0.42 0.50 0.11 0.41 Array 0.25 0.31 0.0 0.31
Throughput B C 0.40 0.37 0.37 0.40 0.40 0.37 0.37 0.40 Throughput B C 0.37 0.26 0.37 0.51 0.26 0.37 0.32 0.28 Throughput B C 0.45 0.42 0.05 0.42 0.44 0.05 0.20 0.42 Throughput B C 0.23 0.23 0.0 0.31 0.34 0.0 0.12 0.31
Table K.4: Mean relative error
C
C
C
C
Latency 0.45 1.06 0.47 0.34 0.25
Array 1.54 0.50 0.47 0.47 Array 0.50 0.58 0.45 0.47 Array 0.35 0.25 0.43 0.08 Array 0.25 0.18 0.38 0.0
D 0.40 0.40 0.40 0.40
A 0.25 0.32 0.33 0.31
D 0.41 0.41 0.41 0.40
A 0.25 0.34 0.29 0.28
D 0.41 0.17 0.42 0.04
A 0.10 0.24 0.24 0.25
D 0.19 0.17 0.34 0.0
A 0.0 0.19 0.18 0.17
D
D
D
D
Latency B C 2.70 1.96 0.59 0.37 0.66 0.37 0.61 0.36 Latency B C 0.33 0.43 0.59 0.49 0.44 0.37 0.49 0.49 Latency B C 0.26 0.34 0.10 0.46 0.34 0.16 0.24 0.47 Latency B C 0.20 0.28 0.0 0.31 0.24 0.0 0.14 0.32
D 3.84 0.60 0.61 0.61 D 0.68 0.81 0.55 0.61 D 0.32 0.36 0.61 0.13 D 0.35 0.19 0.50 0.0
Appendix L
FitnessBuffered model testing on FitnessBuffered samples Application Buffered (sd) Total used
Samples 200
Iters 3
First sample 100 100
Last sample 199
Table L.1: Multiple testing samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
151
152
Overall (FitnessBuffered)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Overall (FitnessBuffered)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Overall (FitnessBuffered) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Overall (FitnessBuffered)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Overall (FitnessBuffered)
Relative error
Figure L.1: The cumulative distribution of relative error over all pairwise predictions.
APPENDIX L. FITNESSBUFFERED MODEL TESTING ON FITNESSBUFFERED SAMPLES
Bandwidth (FitnessBuffered)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Performance Relative Fitness 0.1
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Bandwidth (FitnessBuffered)
0.9
0.8
Relative error Bandwidth (FitnessBuffered)
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute Relative 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Bandwidth (FitnessBuffered)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Bandwidth (FitnessBuffered)
Relative error
Figure L.2: The cumulative distribution of relative error over all pairwise predictions.
153
154
Throughput (FitnessBuffered)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Performance Relative Fitness 0.1
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Throughput (FitnessBuffered)
0.9
0.8
Relative error Throughput (FitnessBuffered)
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute Relative 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Throughput (FitnessBuffered)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Throughput (FitnessBuffered)
Relative error
Figure L.3: The cumulative distribution of relative error over all pairwise predictions.
APPENDIX L. FITNESSBUFFERED MODEL TESTING ON FITNESSBUFFERED SAMPLES
Latency (FitnessBuffered)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Latency (FitnessBuffered)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Latency (FitnessBuffered) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Latency (FitnessBuffered)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Latency (FitnessBuffered)
Relative error
Figure L.4: The cumulative distribution of relative error over all pairwise predictions.
155
156
<-0
[[[0 [0 [0 [1 > .29 0.29 0.01 .26, .53, .80, .07, =1.3 , -0 , 0 4 0 0 1 1 .01 .26 .53) .80) .07) .34) ) )
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
[[0 [0 [0 [0 [1 > .24 0.24 .03, .31, .58, .86, .13, =1.4 ,0 1 .03 0.31 0.58 0.86 1.13 1.41 ) ) ) ) ) )
0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[0 [0 [0 [0 > .27 0.27 0.13 .01, .15, .29, .43, =0.5 , -0 , 0 7 0 0 0 0 .13 .01 .15) .29) .43) .57) ) )
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0
<-0
[[[0 [0 [0 [0 > .28 0.28 0.07 .14, .35, .56, .77, =0.9 , -0 , 0 8 0 0 0 0 .07 .14 .35) .56) .77) .98) ) )
<-0
[[[0 [0 [0 [0 > .31 0.31 0.14 .03, .20, .36, .53, =0.7 , -0 , 0 0 0 0 0 0 .14 .03 .20) .36) .53) .70) ) )
Absolute: Latency (Avg )
<-0
[[[0 [0 [0 [0 > .23 0.23 0.10 .03, .16, .30, .43, =0.5 , -0 , 0 6 0 0 0 0 .10 .03 .16) .30) .43) .56) ) )
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[0 [0 [0 [1 [1 > .24 0.24 .09, .42, .75, .07, .40, =1.7 ,0 3 .09 0.42 0.75 1.07 1.40 1.73 ) ) ) ) ) )
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Fitness: Latency (Avg )
Fraction
Relative Performance: Latency (Avg )
Fraction
Fraction
Relative: Latency (Avg )
[[0 [0 [0 [1 [1 > .27 0.27 .05, .38, .71, .04, .36, =1.6 ,0 9 .05 0.38 0.71 1.04 1.36 1.69 ) ) ) ) ) )
0.1 0.05
Fraction
Fraction
Fraction
0.05
<-0
0.15
Relative Fitness: Throughput (Avg )
0.2
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative: Throughput (Avg ) 0.2
Relative Performance: Throughput (Avg ) 0.25
0
[[[0 [0 [0 [0 > .18 0.18 0.02 .14, .30, .46, .62, =0.7 , -0 , 0 8 0 0 0 0 .02 .14 .30) .46) .62) .78) ) )
0.25 Fraction
Fraction
Fraction [[[0 [0 [0 [0 > .22 0.22 0.08 .06, .20, .34, .48, =0.6 , -0 , 0 2 0 0 0 0 .08 .06 .20) .34) .48) .62) ) )
0.1
<-0
Absolute: Throughput (Avg )
<-0
0.15
0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
Relative Fitness: Bandwidth (Avg ) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Performance: Bandwidth (Avg )
Fraction
Relative: Bandwidth (Avg )
Fraction
Fraction
Absolute: Bandwidth (Avg ) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[0 [0 [0 [0 > .33 0.33 0.10 .12, .35, .57, .79, =1.0 , -0 , 0 2 0 0 0 1 .10 .12 .35) .57) .79) .02) ) )
0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[[0 [0 [0 > .33 0.33 0.17 0.02 .14, .30, .45, =0.6 , -0 , -0 , 0 1 0 0 0 .17 .02 .14 .30) .45) .61) ) ) )
Figure L.5: The probability distribution of relative error over all pairwise predictions.
APPENDIX L. FITNESSBUFFERED MODEL TESTING ON FITNESSBUFFERED SAMPLES
Bandwidth
Fraction
Fraction
Overall 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
, -0
80
)
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
, -0
80
)
Latency
Fraction
Fraction
Throughput 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
157
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
Figure L.6: Probability distributions of the difference in the absolute value of the relative error value−measured value (| predictedmeasured |) between the absolute and relative fitness models, over all predictions. The x value axis represent the amount by which relative error is reduced. Negative values indicate an increase in error.
158
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 4 / 4 / 90 11 / 7 / 80 9 / 3 / 86 2 / 0 / 96 11 / 6 / 81 9 / 5 / 84 6 / 4 / 88 7 / 6 / 85 7 / 5 / 86 1 / 2 / 95 4 / 2 / 92 13 / 8 / 77 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 12 / 4 / 82 2 / 4 / 92 0 / 0 / 98 4 / 7 / 87 3 / 3 / 92 0 / 0 / 98 7 / 7 / 84 8 / 8 / 82 0 / 0 / 98 2 / 5 / 91 10 / 4 / 84 5 / 4 / 89 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 10 / 1 / 87 8 / 8 / 82 9 / 3 / 86 7 / 4 / 87 7 / 6 / 85 9 / 4 / 85 10 / 6 / 82 4 / 8 / 86 10 / 3 / 85 11 / 4 / 83 2 / 4 / 92 8 / 5 / 85 nan% nan% nan%
Bandwidth AM vs. RM RM vs. RM’ 24 / 38 / 36 36 / 46 / 16 33 / 28 / 37 20 / 55 / 23 45 / 39 / 14 35 / 60 / 3 16 / 31 / 51 32 / 46 / 20 28 / 31 / 39 30 / 44 / 24 38 / 33 / 27 24 / 57 / 17 36 / 47 / 15 30 / 49 / 19 28 / 48 / 22 32 / 44 / 22 31 / 39 / 28 40 / 39 / 19 20 / 33 / 45 34 / 49 / 15 30 / 36 / 32 27 / 64 / 7 29 / 25 / 44 39 / 38 / 21 100% 0% 0% 100% 0% 0% Throughput AM vs. RM RM vs. RM’ 21 / 23 / 54 28 / 41 / 29 61 / 37 / 0 31 / 66 / 1 37 / 60 / 1 32 / 35 / 31 54 / 32 / 12 36 / 49 / 13 61 / 37 / 0 35 / 44 / 19 42 / 56 / 0 46 / 50 / 2 45 / 29 / 24 31 / 53 / 14 46 / 41 / 11 37 / 47 / 14 39 / 58 / 1 41 / 28 / 29 57 / 34 / 7 37 / 61 / 0 37 / 32 / 29 28 / 52 / 18 61 / 37 / 0 56 / 40 / 2 100% 0% 0% 100% 0% 0% Latency AM vs. RM RM vs. RM’ 29 / 32 / 37 29 / 44 / 25 33 / 54 / 11 37 / 43 / 18 41 / 33 / 24 24 / 55 / 19 24 / 19 / 55 27 / 62 / 9 33 / 44 / 21 41 / 47 / 10 32 / 28 / 38 23 / 61 / 14 35 / 32 / 31 26 / 52 / 20 37 / 34 / 27 32 / 56 / 10 35 / 47 / 16 43 / 45 / 10 7 / 21 / 70 19 / 61 / 18 31 / 26 / 41 20 / 54 / 24 13 / 65 / 20 53 / 35 / 10 100% 0% 0% 100% 0% 0%
RM’ vs. RF 23 / 59 / 16 40 / 55 / 3 29 / 61 / 8 36 / 56 / 6 43 / 52 / 3 36 / 53 / 9 44 / 46 / 8 38 / 53 / 7 38 / 53 / 7 35 / 53 / 10 26 / 64 / 8 38 / 54 / 6 100% 0% 0%
AM vs. RF 24 / 69 / 5 37 / 54 / 7 29 / 65 / 4 26 / 67 / 5 41 / 53 / 4 17 / 78 / 3 26 / 64 / 8 36 / 57 / 5 40 / 53 / 5 30 / 64 / 4 23 / 70 / 5 45 / 50 / 3 100% 0% 0%
RM’ vs. RF 31 / 58 / 9 41 / 50 / 7 37 / 56 / 5 24 / 67 / 7 28 / 64 / 6 21 / 68 / 9 41 / 53 / 4 44 / 51 / 3 35 / 58 / 5 31 / 62 / 5 22 / 65 / 11 37 / 59 / 2 100% 0% 0%
AM vs. RF 22 / 70 / 6 31 / 62 / 5 27 / 71 / 0 23 / 65 / 10 25 / 67 / 6 15 / 75 / 8 36 / 58 / 4 46 / 46 / 6 28 / 66 / 4 28 / 62 / 8 19 / 73 / 6 43 / 51 / 4 100% 0% 0%
RM’ vs. RF 34 / 59 / 5 37 / 55 / 6 39 / 56 / 3 44 / 48 / 6 42 / 54 / 2 31 / 54 / 13 39 / 54 / 5 52 / 42 / 4 36 / 59 / 3 38 / 52 / 8 31 / 57 / 10 40 / 49 / 9 100% 0% 0%
AM vs. RF 25 / 67 / 6 20 / 75 / 3 27 / 66 / 5 31 / 62 / 5 29 / 65 / 4 27 / 66 / 5 28 / 67 / 3 47 / 47 / 4 40 / 52 / 6 22 / 72 / 4 26 / 68 / 4 27 / 69 / 2 100% 0% 0%
Table L.2: Models are compared across all pairwise predictions. For each prediction, a score of 1 is assigned to the model with the lowest accuracy. If the accuracy of two models is within 0.01, a score of 1 goes to the tie category (3rd number shown). Hypotheses are tested in turn. Specifically, idealized absolute models (i = j) are compared against in-practice absolute models (i 6= j), relative models are compared to absolute models (AM vs. RM), relative models that use performance are compared to those that do not (RM vs. RM’), and relative fitness models are compared to relative models that use performance (RM’ vs. RF). In addition, relative fitness models are compared directly to absolute models (AM vs. RF).
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.20 0.21 0.20 0.15 0.11
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.17 0.17 0.17 0.16
Relative Array A Array B Array C Array D
A 0.17 0.12 0.14 0.13
Relative Performance Array A Array B Array C Array D
A 0.03 0.12 0.12 0.12
Relative Fitness Array A Array B Array C Array D
A 0.0 0.08 0.10 0.08
Bandwidth 0.17 0.19 0.18 0.14 0.10
Pairwise Array A Array 0.18 0.17 0.19 0.16 0.19 0.16 0.18 0.16 Array A Array 0.18 0.16 0.17 0.16 0.20 0.19 0.18 0.17 Array A Array 0.03 0.13 0.13 0.03 0.14 0.15 0.13 0.10 Array A Array 0.0 0.08 0.09 0.0 0.12 0.13 0.10 0.06
Bandwidth B C 0.17 0.18 0.16 0.19 0.19 0.15 0.16 0.19 Bandwidth B C 0.15 0.21 0.16 0.18 0.19 0.15 0.17 0.18 Bandwidth B C 0.12 0.15 0.03 0.15 0.15 0.03 0.09 0.18 Bandwidth B C 0.07 0.13 0.0 0.14 0.11 0.0 0.06 0.14
D 0.22 0.21 0.19 0.19
A 0.18 0.17 0.17 0.17
D 0.24 0.22 0.16 0.19
A 0.18 0.17 0.18 0.20
D 0.16 0.12 0.16 0.03
A 0.03 0.13 0.14 0.16
D 0.10 0.07 0.12 0.0
A 0.0 0.07 0.11 0.10
B
B
B
B
Throughput 0.20 0.20 0.20 0.15 0.10
Array 0.23 0.23 0.22 0.24 Array 0.22 0.21 0.22 0.20 Array 0.16 0.19 0.04 0.21 Array 0.13 0.14 0.0 0.16
Throughput B C 0.16 0.22 0.13 0.23 0.13 0.22 0.15 0.23 Throughput B C 0.15 0.25 0.13 0.25 0.15 0.22 0.16 0.25 Throughput B C 0.11 0.16 0.03 0.22 0.14 0.03 0.10 0.26 Throughput B C 0.06 0.13 0.0 0.13 0.12 0.0 0.05 0.18
Table L.3: Median relative error
C
C
C
C
Latency 0.23 0.25 0.23 0.16 0.13
Array 0.26 0.26 0.25 0.24 Array 0.23 0.24 0.20 0.24 Array 0.16 0.14 0.20 0.03 Array 0.12 0.07 0.15 0.0
D 0.31 0.31 0.31 0.31
A 0.20 0.22 0.22 0.22
D 0.20 0.22 0.21 0.31
A 0.20 0.24 0.23 0.23
D 0.16 0.16 0.22 0.03
A 0.04 0.16 0.17 0.13
D 0.12 0.06 0.15 0.0
A 0.0 0.14 0.14 0.12
D
D
D
D
Latency B C 0.24 0.30 0.20 0.30 0.17 0.29 0.18 0.30 Latency B C 0.18 0.21 0.20 0.21 0.23 0.29 0.20 0.15 Latency B C 0.17 0.17 0.04 0.19 0.17 0.04 0.11 0.21 Latency B C 0.11 0.12 0.0 0.15 0.16 0.0 0.09 0.16
D 0.24 0.24 0.24 0.22 D 0.25 0.29 0.23 0.22 D 0.13 0.17 0.25 0.05 D 0.13 0.09 0.17 0.0
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.33 0.34 0.32 0.23 0.18
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.30 0.30 0.29 0.28
Relative Array A Array B Array C Array D
A 0.30 0.27 0.24 0.26
Relative Performance Array A Array B Array C Array D
A 0.06 0.21 0.15 0.18
Relative Fitness Array A Array B Array C Array D
A 0.0 0.11 0.14 0.11
Bandwidth 0.34 0.34 0.32 0.21 0.16
Pairwise Array A Array 0.29 0.38 0.30 0.36 0.31 0.32 0.30 0.35 Array A Array 0.29 0.34 0.29 0.36 0.28 0.33 0.28 0.33 Array A Array 0.05 0.20 0.21 0.06 0.20 0.21 0.19 0.15 Array A Array 0.0 0.12 0.16 0.0 0.18 0.20 0.14 0.10
Bandwidth B C 0.38 0.31 0.46 0.31 0.34 0.27 0.44 0.32 Bandwidth B C 0.36 0.38 0.46 0.36 0.26 0.27 0.36 0.31 Bandwidth B C 0.18 0.27 0.06 0.23 0.18 0.07 0.15 0.32 Bandwidth B C 0.10 0.26 0.0 0.25 0.17 0.0 0.09 0.21
D 0.36 0.37 0.34 0.34
A 0.29 0.28 0.29 0.29
D 0.42 0.35 0.30 0.34
A 0.29 0.27 0.27 0.29
D 0.21 0.20 0.23 0.07
A 0.05 0.22 0.21 0.23
D 0.15 0.15 0.23 0.0
A 0.0 0.13 0.19 0.16
B
B
B
B
Throughput 0.29 0.30 0.27 0.23 0.17
Array 0.35 0.34 0.32 0.35 Array 0.35 0.33 0.32 0.29 Array 0.25 0.30 0.07 0.31 Array 0.24 0.26 0.0 0.24
Throughput B C 0.21 0.30 0.18 0.30 0.20 0.30 0.20 0.31 Throughput B C 0.19 0.32 0.18 0.32 0.24 0.30 0.20 0.32 Throughput B C 0.13 0.20 0.05 0.33 0.17 0.05 0.12 0.31 Throughput B C 0.10 0.26 0.0 0.19 0.18 0.0 0.08 0.23
Table L.4: Mean relative error
C
C
C
C
Latency 0.35 0.39 0.37 0.26 0.21
Array 0.38 0.37 0.37 0.35 Array 0.35 0.35 0.32 0.35 Array 0.22 0.25 0.27 0.08 Array 0.18 0.16 0.23 0.0
D 0.42 0.42 0.42 0.42
A 0.29 0.32 0.33 0.32
D 0.24 0.30 0.27 0.42
A 0.29 0.34 0.31 0.31
D 0.24 0.24 0.28 0.05
A 0.05 0.21 0.25 0.16
D 0.18 0.14 0.24 0.0
A 0.0 0.24 0.22 0.16
D
D
D
D
Latency B C 0.56 0.44 0.43 0.42 0.44 0.39 0.41 0.43 Latency B C 0.47 0.37 0.43 0.32 0.48 0.39 0.42 0.23 Latency B C 0.29 0.27 0.06 0.34 0.28 0.08 0.19 0.29 Latency B C 0.16 0.21 0.0 0.33 0.24 0.0 0.13 0.28
D 0.36 0.32 0.36 0.30 D 0.39 0.40 0.40 0.30 D 0.21 0.31 0.31 0.12 D 0.20 0.18 0.23 0.0
Appendix M
FitnessFS model testing on FitnessFS samples Application FS (fs) Total used
Samples 200
Iters 3
First sample 100 100
Last sample 199
Table M.1: Multiple testing samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
161
162
Overall (FitnessFS)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Overall (FitnessFS)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Overall (FitnessFS) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Overall (FitnessFS)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Overall (FitnessFS)
Relative error
Figure M.1: The cumulative distribution of relative error over all pairwise predictions.
163
APPENDIX M. FITNESSFS MODEL TESTING ON FITNESSFS SAMPLES
Bandwidth (FitnessFS)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Bandwidth (FitnessFS)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Bandwidth (FitnessFS) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Bandwidth (FitnessFS)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Bandwidth (FitnessFS)
Relative error
Figure M.2: The cumulative distribution of relative error over all pairwise predictions.
164
Throughput (FitnessFS)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Throughput (FitnessFS)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Throughput (FitnessFS) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Throughput (FitnessFS)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Throughput (FitnessFS)
Relative error
Figure M.3: The cumulative distribution of relative error over all pairwise predictions.
165
APPENDIX M. FITNESSFS MODEL TESTING ON FITNESSFS SAMPLES
Latency (FitnessFS)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Latency (FitnessFS)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Latency (FitnessFS) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Latency (FitnessFS)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Latency (FitnessFS)
Relative error
Figure M.4: The cumulative distribution of relative error over all pairwise predictions.
166
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[0 [0 [0 [0 [1 > .21 0.21 .05, .31, .57, .83, .09, =1.3 ,0 5 .05 0.31 0.57 0.83 1.09 1.35 ) ) ) ) ) )
<-0
[[[0 [0 [0 [0 > .27 0.27 0.02 .22, .47, .71, .96, =1.2 , -0 , 0 1 0 0 0 1 .02 .22 .47) .71) .96) .21) ) )
<-0
[[[0 [0 [0 [0 > .21 0.21 0.10 .00, .10, .21, .31, =0.4 , -0 , 0 1 0 0 0 0 .10 .00 .10) .21) .31) .41) ) )
<-0
[[[[0 [0 [0 > .31 0.31 0.16 0.00 .15, .30, .46, =0.6 , -0 , -0 , 0 1 0 0 0 .16 .00 .15 .30) .46) .61) ) ) )
0.1
0
<-0
[[[[0 [0 [0 > .24 0.24 0.12 0.01 .11, .22, .34, =0.4 , -0 , -0 , 0 6 0 0 0 .12 .01 .11 .22) .34) .46) ) ) )
<-0
[[[[0 [0 [0 > .34 0.34 0.19 0.04 .11, .26, .41, =0.5 , -0 , -0 , 0 6 0 0 0 .19 .04 .11 .26) .41) .56) ) ) )
<-0
[[[[0 [0 [0 > .22 0.22 0.12 0.03 .07, .16, .26, =0.3 , -0 , -0 , 0 5 0 0 0 .12 .03 .07 .16) .26) .35) ) ) )
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[0 [0 [0 [0 > .31 0.31 0.11 .10, .30, .51, .71, =0.9 , -0 , 0 1 0 0 0 0 .11 .10 .30) .51) .71) .91) ) )
0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Fitness: Latency (Avg )
Fraction
Relative Performance: Latency (Avg )
Fraction
Fraction
Fraction
Fraction
Fraction
0.15
0.05
[[[0 [0 [0 [0 > .32 0.32 0.11 .10, .31, .53, .74, =0.9 , -0 , 0 5 0 0 0 0 .11 .10 .31) .53) .74) .95) ) )
0.3 0.25 0.2 0.15 0.1 0.05 0
Absolute: Latency (Avg )
0.2
<-0
[[[0 [0 [0 [0 > .23 0.23 0.09 .05, .20, .34, .48, =0.6 , -0 , 0 2 0 0 0 0 .09 .05 .20) .34) .48) .62) ) )
Relative Fitness: Throughput (Avg ) 0.25
Relative: Latency (Avg ) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
Relative: Throughput (Avg )
Fraction
0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Performance: Throughput (Avg ) 0.3 0.25 0.2 0.15 0.1 0.05 0
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Absolute: Throughput (Avg )
Fraction
Fraction
Relative Fitness: Bandwidth (Avg ) 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Performance: Bandwidth (Avg )
Fraction
Relative: Bandwidth (Avg )
Fraction
Fraction
Absolute: Bandwidth (Avg ) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[[0 [0 [0 > .36 0.36 0.19 0.01 .17, .34, .52, =0.6 , -0 , -0 , 0 9 0 0 0 .19 .01 .17 .34) .52) .69) ) ) )
0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[0 [0 [0 [0 > .31 0.31 0.11 .10, .30, .50, .70, =0.9 , -0 , 0 0 0 0 0 0 .11 .10 .30) .50) .70) .90) ) )
Figure M.5: The probability distribution of relative error over all pairwise predictions.
APPENDIX M. FITNESSFS MODEL TESTING ON FITNESSFS SAMPLES
Bandwidth
Fraction
Fraction
Overall 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
, -0
80
)
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
, -0
80
)
Latency
Fraction
Fraction
Throughput 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
167
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
Figure M.6: Probability distributions of the difference in the absolute value of the relative error value−measured value (| predictedmeasured |) between the absolute and relative fitness models, over all predictions. The x value axis represent the amount by which relative error is reduced. Negative values indicate an increase in error.
168
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 11 / 7 / 80 19 / 6 / 73 19 / 19 / 60 11 / 14 / 73 7 / 5 / 86 11 / 21 / 66 10 / 14 / 74 10 / 13 / 75 14 / 16 / 68 12 / 13 / 73 9 / 9 / 80 20 / 5 / 73 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 2 / 1 / 95 5 / 3 / 90 0 / 0 / 98 18 / 14 / 66 3 / 3 / 92 0 / 0 / 98 20 / 12 / 66 5 / 2 / 91 0 / 0 / 98 19 / 14 / 65 1 / 1 / 96 7 / 2 / 89 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 5 / 4 / 89 21 / 5 / 72 16 / 8 / 74 5 / 0 / 93 14 / 6 / 78 8 / 9 / 81 9 / 4 / 85 4 / 3 / 91 12 / 6 / 80 8 / 3 / 87 7 / 4 / 87 27 / 7 / 64 nan% nan% nan%
Bandwidth AM vs. RM RM vs. RM’ 22 / 32 / 44 34 / 53 / 11 35 / 42 / 21 28 / 58 / 12 41 / 30 / 27 31 / 59 / 8 29 / 34 / 35 46 / 48 / 4 29 / 36 / 33 23 / 41 / 34 47 / 30 / 21 30 / 58 / 10 49 / 36 / 13 41 / 42 / 15 39 / 33 / 26 40 / 42 / 16 52 / 34 / 12 33 / 38 / 27 57 / 38 / 3 25 / 64 / 9 37 / 35 / 26 20 / 63 / 15 43 / 38 / 17 27 / 67 / 4 100% 0% 0% 100% 0% 0% Throughput AM vs. RM RM vs. RM’ 38 / 54 / 6 36 / 46 / 16 61 / 35 / 2 0 / 0 / 98 36 / 61 / 1 34 / 58 / 6 36 / 23 / 39 29 / 55 / 14 63 / 33 / 2 32 / 58 / 8 35 / 63 / 0 28 / 62 / 8 28 / 25 / 45 38 / 39 / 21 35 / 60 / 3 41 / 31 / 26 33 / 63 / 2 47 / 46 / 5 36 / 18 / 44 22 / 63 / 13 51 / 46 / 1 31 / 58 / 9 60 / 36 / 2 28 / 63 / 7 100% 0% 0% 100% 0% 0% Latency AM vs. RM RM vs. RM’ 17 / 25 / 56 48 / 41 / 9 24 / 27 / 47 39 / 38 / 21 18 / 33 / 47 29 / 39 / 30 34 / 37 / 27 34 / 51 / 13 32 / 20 / 46 24 / 42 / 32 28 / 29 / 41 23 / 41 / 34 28 / 44 / 26 34 / 51 / 13 38 / 39 / 21 42 / 37 / 19 17 / 22 / 59 28 / 34 / 36 31 / 45 / 22 34 / 54 / 10 42 / 35 / 21 27 / 53 / 18 37 / 38 / 23 34 / 47 / 17 100% 0% 0% 100% 0% 0%
RM’ vs. RF 33 / 57 / 8 39 / 50 / 9 23 / 64 / 11 27 / 64 / 7 49 / 44 / 5 36 / 53 / 9 49 / 44 / 5 37 / 54 / 7 30 / 58 / 10 33 / 53 / 12 39 / 47 / 12 41 / 50 / 7 100% 0% 0%
AM vs. RF 31 / 62 / 5 30 / 65 / 3 25 / 68 / 5 41 / 54 / 3 38 / 57 / 3 22 / 70 / 6 47 / 44 / 7 32 / 59 / 7 28 / 64 / 6 31 / 58 / 9 22 / 70 / 6 26 / 67 / 5 100% 0% 0%
RM’ vs. RF 39 / 50 / 9 27 / 69 / 2 30 / 56 / 12 31 / 52 / 15 38 / 53 / 7 39 / 50 / 9 47 / 41 / 10 36 / 56 / 6 39 / 49 / 10 42 / 47 / 9 29 / 60 / 9 44 / 45 / 9 100% 0% 0%
AM vs. RF 35 / 59 / 4 32 / 60 / 6 22 / 74 / 2 27 / 61 / 10 38 / 54 / 6 17 / 79 / 2 41 / 52 / 5 43 / 51 / 4 23 / 73 / 2 34 / 56 / 8 26 / 65 / 7 42 / 50 / 6 100% 0% 0%
RM’ vs. RF 49 / 41 / 8 31 / 60 / 7 40 / 54 / 4 30 / 63 / 5 51 / 42 / 5 38 / 52 / 8 34 / 56 / 8 43 / 51 / 4 42 / 51 / 5 30 / 57 / 11 37 / 53 / 8 50 / 41 / 7 100% 0% 0%
AM vs. RF 51 / 43 / 4 38 / 53 / 7 34 / 60 / 4 27 / 71 / 0 55 / 39 / 4 27 / 65 / 6 28 / 70 / 0 48 / 44 / 6 48 / 45 / 5 27 / 68 / 3 30 / 62 / 6 47 / 46 / 5 100% 0% 0%
Table M.2: Models are compared across all pairwise predictions. For each prediction, a score of 1 is assigned to the model with the lowest accuracy. If the accuracy of two models is within 0.01, a score of 1 goes to the tie category (3rd number shown). Hypotheses are tested in turn. Specifically, idealized absolute models (i = j) are compared against in-practice absolute models (i 6= j), relative models are compared to absolute models (AM vs. RM), relative models that use performance are compared to those that do not (RM vs. RM’), and relative fitness models are compared to relative models that use performance (RM’ vs. RF). In addition, relative fitness models are compared directly to absolute models (AM vs. RF).
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.15 0.16 0.15 0.12 0.10
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.15 0.14 0.13 0.15
Relative Array A Array B Array C Array D
A 0.15 0.13 0.14 0.21
Relative Performance Array A Array B Array C Array D
A 0.03 0.11 0.13 0.10
Relative Fitness Array A Array B Array C Array D
A 0.0 0.07 0.11 0.08
Bandwidth 0.17 0.17 0.16 0.11 0.09
Pairwise Array A Array 0.17 0.13 0.17 0.12 0.17 0.12 0.17 0.13 Array A Array 0.17 0.12 0.15 0.12 0.15 0.13 0.17 0.14 Array A Array 0.03 0.10 0.12 0.02 0.14 0.12 0.12 0.09 Array A Array 0.0 0.09 0.09 0.0 0.11 0.10 0.09 0.07
Bandwidth B C 0.15 0.21 0.14 0.15 0.14 0.14 0.15 0.24 Bandwidth B C 0.14 0.20 0.14 0.15 0.15 0.14 0.14 0.21 Bandwidth B C 0.10 0.11 0.02 0.10 0.14 0.02 0.07 0.11 Bandwidth B C 0.06 0.10 0.0 0.11 0.08 0.0 0.07 0.12
D 0.17 0.15 0.18 0.20
A 0.10 0.12 0.13 0.12
D 0.20 0.17 0.18 0.20
A 0.10 0.12 0.13 0.15
D 0.14 0.09 0.16 0.04
A 0.02 0.08 0.10 0.10
D 0.07 0.08 0.11 0.0
A 0.0 0.07 0.11 0.09
B
B
B
B
Throughput 0.15 0.15 0.14 0.11 0.09
Array 0.16 0.13 0.13 0.15 Array 0.17 0.16 0.13 0.17 Array 0.14 0.12 0.02 0.13 Array 0.10 0.12 0.0 0.14
Throughput B C 0.10 0.13 0.10 0.12 0.10 0.12 0.10 0.13 Throughput B C 0.10 0.17 0.10 0.17 0.10 0.12 0.15 0.17 Throughput B C 0.08 0.17 0.02 0.11 0.11 0.01 0.09 0.12 Throughput B C 0.07 0.10 0.0 0.10 0.10 0.0 0.05 0.12
Table M.3: Median relative error
C
C
C
C
Latency 0.15 0.17 0.16 0.14 0.12
Array 0.21 0.17 0.21 0.21 Array 0.17 0.14 0.14 0.21 Array 0.13 0.09 0.14 0.05 Array 0.09 0.08 0.12 0.0
D 0.27 0.27 0.27 0.27
A 0.25 0.27 0.27 0.27
D 0.15 0.13 0.13 0.27
A 0.25 0.24 0.23 0.22
D 0.12 0.09 0.14 0.21
A 0.03 0.23 0.20 0.15
D 0.10 0.08 0.12 0.0
A 0.0 0.13 0.13 0.10
D
D
D
D
Latency B C 0.12 0.17 0.12 0.13 0.12 0.13 0.14 0.14 Latency B C 0.12 0.16 0.12 0.16 0.15 0.13 0.13 0.16 Latency B C 0.13 0.17 0.03 0.13 0.14 0.04 0.10 0.15 Latency B C 0.16 0.10 0.0 0.15 0.14 0.0 0.08 0.16
D 0.15 0.14 0.14 0.13 D 0.15 0.13 0.14 0.13 D 0.14 0.09 0.14 0.03 D 0.09 0.07 0.15 0.0
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.25 0.26 0.25 0.19 0.16
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.21 0.19 0.20 0.19
Relative Array A Array B Array C Array D
A 0.21 0.21 0.26 0.29
Relative Performance Array A Array B Array C Array D
A 0.07 0.17 0.17 0.15
Relative Fitness Array A Array B Array C Array D
A 0.0 0.11 0.17 0.10
Bandwidth 0.29 0.31 0.29 0.18 0.13
Pairwise Array A Array 0.21 0.24 0.21 0.23 0.23 0.23 0.21 0.26 Array A Array 0.21 0.22 0.22 0.23 0.25 0.22 0.24 0.24 Array A Array 0.06 0.16 0.19 0.06 0.20 0.22 0.18 0.13 Array A Array 0.0 0.16 0.15 0.0 0.18 0.19 0.14 0.10
Bandwidth B C 0.32 0.40 0.31 0.36 0.28 0.33 0.37 0.40 Bandwidth B C 0.27 0.31 0.31 0.27 0.28 0.33 0.28 0.34 Bandwidth B C 0.17 0.19 0.09 0.19 0.24 0.06 0.12 0.19 Bandwidth B C 0.12 0.14 0.0 0.17 0.15 0.0 0.10 0.16
D 0.30 0.31 0.35 0.31
A 0.17 0.17 0.17 0.17
D 0.33 0.33 0.35 0.31
A 0.17 0.17 0.17 0.18
D 0.19 0.16 0.25 0.06
A 0.03 0.12 0.17 0.13
D 0.11 0.10 0.15 0.0
A 0.0 0.10 0.16 0.11
B
B
B
B
Throughput 0.23 0.23 0.20 0.16 0.13
Array 0.31 0.28 0.27 0.30 Array 0.27 0.27 0.27 0.28 Array 0.22 0.21 0.05 0.20 Array 0.16 0.20 0.0 0.20
Throughput B C 0.19 0.25 0.18 0.24 0.20 0.25 0.18 0.25 Throughput B C 0.14 0.23 0.18 0.23 0.18 0.25 0.20 0.23 Throughput B C 0.13 0.23 0.04 0.19 0.18 0.03 0.12 0.16 Throughput B C 0.11 0.13 0.0 0.14 0.18 0.0 0.08 0.16
Table M.4: Mean relative error
C
C
C
C
Latency 0.22 0.25 0.25 0.22 0.21
Array 0.29 0.28 0.30 0.28 Array 0.26 0.27 0.27 0.28 Array 0.19 0.18 0.21 0.13 Array 0.12 0.12 0.17 0.0
D 0.31 0.31 0.31 0.31
A 0.27 0.29 0.30 0.28
D 0.23 0.29 0.20 0.31
A 0.27 0.27 0.31 0.24
D 0.17 0.20 0.18 0.25
A 0.07 0.28 0.27 0.25
D 0.12 0.10 0.16 0.0
A 0.0 0.24 0.21 0.19
D
D
D
D
Latency B C 0.22 0.28 0.20 0.24 0.22 0.22 0.22 0.26 Latency B C 0.23 0.28 0.20 0.31 0.20 0.22 0.23 0.27 Latency B C 0.19 0.23 0.05 0.24 0.23 0.06 0.14 0.24 Latency B C 0.24 0.21 0.0 0.29 0.24 0.0 0.13 0.27
D 0.24 0.20 0.23 0.20 D 0.24 0.20 0.27 0.20 D 0.22 0.18 0.20 0.09 D 0.14 0.14 0.21 0.0
Appendix N
FitnessCache model testing on FitnessCache samples Application Cache (cache) Total used
Samples 70
Iters 3
First sample 35 35
Last sample 69
Table N.1: Multiple testing samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
171
172
Overall (FitnessCache)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error Overall (FitnessCache)
0.9
0.9
0.8
0.8
0.7
0.7
0.6
0.6
Fraction
0.5 0.4 0.3
0.4 0.3 0.2
Relative error
0.9
1 1
0.8 0.8
0.7
Relative Performance Relative Fitness 0.6
1
0.9
0.8
0.7
0.6
0.5
0.3
0
0.4
Relative Relative Performance
0.1
0.5
0.2
0.4
0.3
0.3
0.4
0.2
0.5
0.1
Fraction
0.6
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.9
0.7
0.5
0.4
0.3
Relative error Overall (FitnessCache)
0.7
0.2
0.2
Relative error Overall (FitnessCache) 0.8
0
0.1
0
1
0
0.9
0.1
Absolute Relative
0.1 0.9
0.8
0.7
0.6
0.4
0.3
0.2
0
0.1
0
0.5
Absolute (i=j) Absolute
0.1
0.6
0.2
Fraction
0.5
0
Fraction
Overall (FitnessCache)
Relative error
Figure N.1: The cumulative distribution of relative error over all pairwise predictions.
173
APPENDIX N. FITNESSCACHE MODEL TESTING ON FITNESSCACHE SAMPLES
Bandwidth (FitnessCache)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error Bandwidth (FitnessCache)
0.7
0.7
0.6
0.6
0.5
0.5 Fraction
0.4 0.3
0.2 0.1
Relative error
0.9
1 1
0.8
0.9
0.7
0.5
0.4
0.8
0.7
Relative Performance Relative Fitness 0.6
1
0.9
0.8
0.7
0.6
0.5
0.3
0
0.4
Relative Relative Performance
0.1
0.5
0.2
0.4
0.3
0.3
0.4
0.2
Fraction
0.5
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.1
0.6
0.2
0.3
Relative error Bandwidth (FitnessCache)
0.7
0
0.2
0
1
Relative error Bandwidth (FitnessCache) 0.8
0.1
Absolute Relative
0 0.9
0.8
0.7
0.6
0.4
0.3
0.2
0
0.1
0
0.5
Absolute (i=j) Absolute
0.1
0.1
Fraction
0.3
0.6
0.2
0.4
0
Fraction
Bandwidth (FitnessCache)
Relative error
Figure N.2: The cumulative distribution of relative error over all pairwise predictions.
174
Throughput (FitnessCache)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error Throughput (FitnessCache)
0.9
0.9
0.8
0.8
0.7
0.7
0.6
0.6
Fraction
0.5 0.4 0.3
0.4 0.3 0.2
Relative error
1 1
0.8
Relative Performance Relative Fitness 0.7
1
0.9
0.8
0.7
0.6
0.5
0.3
0.2
0
0.4
Relative Relative Performance
0.1
0.6
0.2
0.5
0.3
0.4
0.4
0.3
0.5
0.2
Fraction
0.6
0.1
0.7
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.9
0.8
Relative error Throughput (FitnessCache)
0.9
0.7
0.5
0.4
0.3
0.2
Relative error Throughput (FitnessCache) 0.8
0
0.1
0
1
0
0.9
0.1
Absolute Relative
0.1 0.9
0.8
0.7
0.6
0.4
0.3
0.2
0
0.1
0
0.5
Absolute (i=j) Absolute
0.1
0.6
0.2
Fraction
0.5
0
Fraction
Throughput (FitnessCache)
Relative error
Figure N.3: The cumulative distribution of relative error over all pairwise predictions.
175
APPENDIX N. FITNESSCACHE MODEL TESTING ON FITNESSCACHE SAMPLES
Latency (FitnessCache)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Latency (FitnessCache)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Latency (FitnessCache) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Latency (FitnessCache)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Latency (FitnessCache)
Relative error
Figure N.4: The cumulative distribution of relative error over all pairwise predictions.
176
<1
[ [ [ [ [ [ > .12 1.12 3.13 5.14 7.14 9.15 11.1 =13 ,3 , , , , 5 . .13 5.14 7.14 9.15 11.1 , 13 16 .16 ) ) ) ) 5) )
0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
<1
[ [ [ [ [ [ > .43 1.43 3.72 6.01 8.30 10.6 12.8 =15 ,3 , , , 0 9 . .72 6.01 8.30 10.6 , 12 , 15 18 .89 .18 ) ) ) 0) ) )
<-0
[[[0 [0 [0 [0 > .37 0.37 0.16 .04, .24, .45, .65, =0.8 , -0 , 0 5 0 0 0 0 .16 .04 .24) .45) .65) .85) ) )
<-0
[[0 [0 [1 [1 [1 > .07 0.07 .32, .70, .09, .48, .87, =2.2 ,0 6 .32 0.70 1.09 1.48 1.87 2.26 ) ) ) ) ) )
0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
[0.
[ [ [ [ > .11 39 0.90 1.41 1.91 2.42 =2.9 , 0 , 0. , 1. , 1. , 2. , 2. 3 41 91 42 93 .39 90 ) ) ) ) ) )
[[[0 [0 [0 [0 > .31 0.31 0.07 .16, .40, .64, .87, =1.1 , -0 , 0 1 0 0 0 1 .07 .16 .40) .64) .87) .11) ) )
0.2 0
<-0
[[0 [0 [1 [1 [2 > .20 0.20 .28, .76, .24, .72, .20, =2.6 ,0 8 .28 0.76 1.24 1.72 2.20 2.68 ) ) ) ) ) )
Relative Fitness: Latency (Avg )
Fraction
Fraction
Fraction .12
[ [ [ [ > .12 36 0.85 1.33 1.81 2.29 =2.7 , 0 , 0. , 1. , 1. , 2. , 2. 7 33 81 29 77 .36 85 ) ) ) ) ) )
0.3
Relative Performance: Latency (Avg )
0.4 0.3 0.2 0.1
[0.
0.4
0.1 <-0
0.5
[-0
[[0 [0 [0 [1 [1 > .20 0.20 .18, .57, .95, .34, .73, =2.1 ,0 1 .18 0.57 0.95 1.34 1.73 2.11 ) ) ) ) ) )
Absolute: Latency (Avg )
0.6
<-0
<-0
0.5
Relative: Latency (Avg ) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.6 Fraction
Fraction
.11
[-0
[ [ [ [ [ [ > .52 0.52 1.74 2.97 4.20 5.43 6.66 =7.8 ,1 , , , , , .74 2.97 4.20 5.43 6.66 7.88 8 ) ) ) ) ) )
Relative Fitness: Throughput (Avg )
Fraction <-0
<0
Relative: Throughput (Avg )
Fraction
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Performance: Throughput (Avg ) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Absolute: Throughput (Avg )
Fraction
Fraction
Relative Fitness: Bandwidth (Avg ) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Performance: Bandwidth (Avg )
Fraction
Relative: Bandwidth (Avg )
Fraction
Fraction
Absolute: Bandwidth (Avg ) 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0
<-0
[[0 [0 [1 [1 [2 > .20 0.20 .28, .76, .24, .72, .20, =2.6 ,0 8 .28 0.76 1.24 1.72 2.20 2.68 ) ) ) ) ) )
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[0 [0 [0 [0 > .48 0.48 0.24 .01, .26, .50, .75, =1.0 , -0 , 0 0 0 0 0 1 .24 .01 .26) .50) .75) .00) ) )
Figure N.5: The probability distribution of relative error over all pairwise predictions.
APPENDIX N. FITNESSCACHE MODEL TESTING ON FITNESSCACHE SAMPLES
Overall
Bandwidth
0.25 Fraction
Fraction
0.2 0.15 0.1 0.05 0
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
, -0
80
)
Throughput
Fraction
Fraction
0.2
0.1 0.05 0
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
, -0
80
)
Latency
0.25
0.15
177
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
Figure N.6: Probability distributions of the difference in the absolute value of the relative error value−measured value (| predictedmeasured |) between the absolute and relative fitness models, over all predictions. The x value axis represent the amount by which relative error is reduced. Negative values indicate an increase in error.
178
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 1 / 1 / 31 0 / 0 / 33 0 / 0 / 33 1 / 0 / 32 0 / 0 / 33 0 / 0 / 33 1 / 0 / 32 1 / 2 / 30 0 / 0 / 33 1 / 0 / 32 1 / 2 / 30 0 / 0 / 33 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 1 / 0 / 32 0 / 0 / 33 0 / 2 / 31 1 / 0 / 32 0 / 0 / 33 0 / 2 / 31 2 / 0 / 31 1 / 1 / 31 0 / 2 / 31 3 / 0 / 30 3 / 1 / 29 0 / 0 / 33 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 3 / 1 / 29 6 / 1 / 26 1 / 0 / 32 0 / 0 / 33 5 / 1 / 27 1 / 0 / 32 0 / 0 / 33 2 / 3 / 28 0 / 0 / 33 2 / 2 / 29 4 / 4 / 25 1 / 1 / 31 nan% nan% nan%
Bandwidth AM vs. RM RM vs. RM’ 18 / 15 / 0 7 / 26 / 0 0 / 0 / 33 2 / 0 / 31 2 / 2 / 29 10 / 15 / 8 14 / 17 / 2 19 / 13 / 1 0 / 0 / 33 5 / 9 / 19 2 / 2 / 29 6 / 17 / 10 0 / 0 / 33 14 / 16 / 3 18 / 15 / 0 7 / 26 / 0 0 / 0 / 33 3 / 23 / 7 1 / 7 / 25 11 / 9 / 13 8 / 17 / 8 11 / 20 / 2 0 / 0 / 33 10 / 18 / 5 100% 0% 0% 100% 0% 0% Throughput AM vs. RM RM vs. RM’ 0 / 2 / 31 4 / 7 / 22 1 / 6 / 26 23 / 9 / 1 0 / 0 / 33 6 / 12 / 15 5 / 7 / 21 10 / 10 / 13 14 / 18 / 1 9 / 17 / 7 3 / 2 / 28 25 / 7 / 1 5 / 3 / 25 4 / 8 / 21 2 / 6 / 25 15 / 16 / 2 0 / 0 / 33 4 / 10 / 19 3 / 3 / 27 7 / 8 / 18 15 / 10 / 8 16 / 15 / 2 0 / 0 / 33 5 / 8 / 20 100% 0% 0% 100% 0% 0% Latency AM vs. RM RM vs. RM’ 4 / 5 / 24 8 / 3 / 22 7 / 7 / 19 4 / 11 / 18 3 / 2 / 28 5 / 3 / 25 0 / 0 / 33 7 / 17 / 9 2 / 5 / 26 3 / 11 / 19 0 / 1 / 32 0 / 4 / 29 4 / 8 / 21 2 / 0 / 31 3 / 2 / 28 6 / 8 / 19 0 / 0 / 33 5 / 7 / 21 3 / 7 / 23 4 / 2 / 27 5 / 3 / 25 5 / 8 / 20 3 / 3 / 27 16 / 9 / 8 100% 0% 0% 100% 0% 0%
RM’ vs. RF 3 / 30 / 0 7 / 26 / 0 9 / 24 / 0 12 / 21 / 0 11 / 22 / 0 7 / 25 / 1 6 / 26 / 1 2 / 31 / 0 10 / 21 / 2 2 / 31 / 0 5 / 28 / 0 7 / 24 / 2 100% 0% 0%
AM vs. RF 2 / 31 / 0 8 / 25 / 0 10 / 23 / 0 12 / 20 / 1 14 / 19 / 0 7 / 26 / 0 4 / 28 / 1 3 / 30 / 0 4 / 28 / 1 5 / 28 / 0 4 / 28 / 1 8 / 22 / 3 100% 0% 0%
RM’ vs. RF 7 / 24 / 2 3 / 30 / 0 9 / 23 / 1 9 / 20 / 4 17 / 16 / 0 5 / 28 / 0 10 / 20 / 3 13 / 18 / 2 7 / 26 / 0 9 / 23 / 1 7 / 24 / 2 8 / 22 / 3 100% 0% 0%
AM vs. RF 5 / 27 / 1 7 / 25 / 1 8 / 25 / 0 10 / 21 / 2 17 / 16 / 0 8 / 25 / 0 11 / 20 / 2 11 / 22 / 0 7 / 25 / 1 9 / 23 / 1 9 / 23 / 1 9 / 23 / 1 100% 0% 0%
RM’ vs. RF 11 / 18 / 4 8 / 25 / 0 12 / 17 / 4 8 / 24 / 1 15 / 16 / 2 12 / 17 / 4 14 / 18 / 1 17 / 15 / 1 15 / 15 / 3 9 / 22 / 2 20 / 9 / 4 13 / 19 / 1 100% 0% 0%
AM vs. RF 9 / 22 / 2 5 / 27 / 1 13 / 16 / 4 8 / 24 / 1 13 / 20 / 0 9 / 20 / 4 12 / 21 / 0 18 / 15 / 0 16 / 14 / 3 9 / 21 / 3 21 / 9 / 3 17 / 14 / 2 100% 0% 0%
Table N.2: Models are compared across all pairwise predictions. For each prediction, a score of 1 is assigned to the model with the lowest accuracy. If the accuracy of two models is within 0.01, a score of 1 goes to the tie category (3rd number shown). Hypotheses are tested in turn. Specifically, idealized absolute models (i = j) are compared against in-practice absolute models (i 6= j), relative models are compared to absolute models (AM vs. RM), relative models that use performance are compared to those that do not (RM vs. RM’), and relative fitness models are compared to relative models that use performance (RM’ vs. RF). In addition, relative fitness models are compared directly to absolute models (AM vs. RF).
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.39 0.40 0.38 0.34 0.14
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.42 0.42 0.42 0.42
Relative Array A Array B Array C Array D
A 0.42 0.82 0.42 0.39
Relative Performance Array A Array B Array C Array D
A 0.19 0.55 0.49 0.45
Relative Fitness Array A Array B Array C Array D
A 0.0 0.14 0.08 0.07
Bandwidth 0.65 0.67 0.70 0.47 0.14
Pairwise Array A Array 0.34 0.35 0.34 0.33 0.34 0.33 0.34 0.33 Array A Array 0.34 0.29 0.33 0.33 0.34 0.27 0.33 0.32 Array A Array 0.16 0.25 0.32 0.16 0.33 0.28 0.31 0.31 Array A Array 0.0 0.08 0.11 0.0 0.14 0.17 0.10 0.14
Bandwidth B C 1.37 0.63 1.37 0.63 0.91 0.63 0.88 0.63 Bandwidth B C 0.79 0.63 1.37 0.63 0.79 0.63 0.83 0.63 Bandwidth B C 0.46 0.63 0.25 0.47 0.53 0.28 0.58 0.30 Bandwidth B C 0.07 0.26 0.0 0.33 0.16 0.0 0.08 0.11
D 0.62 0.62 0.62 0.62
A 0.28 0.28 0.29 0.29
D 0.62 0.62 0.62 0.62
A 0.28 0.23 0.30 0.29
D 0.45 0.36 0.25 0.23
A 0.16 0.28 0.27 0.27
D 0.23 0.06 0.12 0.0
A 0.0 0.09 0.13 0.08
B
B
B
B
Throughput 0.36 0.36 0.36 0.36 0.13
Array 0.55 0.53 0.50 0.47 Array 0.46 0.46 0.50 0.50 Array 0.55 0.44 0.25 0.44 Array 0.18 0.33 0.0 0.13
Throughput B C 0.32 0.43 0.32 0.43 0.33 0.43 0.33 0.43 Throughput B C 0.29 0.27 0.32 0.43 0.27 0.43 0.38 0.43 Throughput B C 0.26 0.83 0.23 0.40 0.28 0.19 0.35 0.40 Throughput B C 0.08 0.10 0.0 0.31 0.17 0.0 0.07 0.12
Table N.3: Median relative error
C
C
C
C
Latency 0.18 0.20 0.18 0.17 0.14
Array 0.41 0.41 0.41 0.42 Array 0.42 0.42 0.41 0.42 Array 0.37 0.42 0.25 0.19 Array 0.13 0.09 0.12 0.0
D 0.45 0.45 0.45 0.47
A 0.25 0.25 0.25 0.31
D 0.45 0.45 0.45 0.47
A 0.25 0.25 0.22 0.21
D 0.41 0.72 0.35 0.32
A 0.10 0.14 0.22 0.19
D 0.10 0.13 0.12 0.0
A 0.0 0.08 0.15 0.14
D
D
D
D
Latency B C 0.13 0.47 0.13 0.47 0.13 0.40 0.13 0.39 Latency B C 0.11 0.43 0.13 0.40 0.13 0.40 0.14 0.40 Latency B C 0.12 0.40 0.07 0.33 0.13 0.24 0.11 0.49 Latency B C 0.06 0.13 0.0 0.33 0.16 0.0 0.16 0.21
D 0.13 0.13 0.13 0.13 D 0.13 0.13 0.13 0.13 D 0.12 0.11 0.14 0.07 D 0.10 0.07 0.12 0.0
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 1.00 1.03 1.12 0.87 0.27
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.99 1.09 1.09 1.09
Relative Array A Array B Array C Array D
A 0.99 1.41 1.09 1.05
Relative Performance Array A Array B Array C Array D
A 0.82 1.59 0.93 0.99
Relative Fitness Array A Array B Array C Array D
A 0.0 0.31 0.19 0.17
Bandwidth 2.11 2.10 2.43 1.37 0.26
Pairwise Array A Array 0.56 1.14 0.60 1.14 0.60 1.11 0.61 1.07 Array A Array 0.56 1.81 0.70 1.14 0.58 1.80 0.58 0.96 Array A Array 0.42 0.54 0.71 0.28 0.50 0.52 0.53 0.71 Array A Array 0.0 0.15 0.20 0.0 0.21 0.25 0.22 0.26
Bandwidth B C 2.90 1.68 2.95 1.68 2.86 1.68 2.70 1.68 Bandwidth B C 4.97 1.68 2.95 1.68 4.97 1.68 2.20 1.68 Bandwidth B C 1.12 2.56 0.51 2.48 1.05 0.71 1.39 0.96 Bandwidth B C 0.15 0.31 0.0 0.53 0.20 0.0 0.24 0.29
D 2.81 2.81 2.81 2.81
A 0.37 0.39 0.40 0.42
D 2.81 2.81 2.81 2.81
A 0.37 0.37 0.39 0.40
D 1.40 1.02 0.92 0.90
A 0.23 0.30 0.26 0.28
D 0.31 0.18 0.16 0.0
A 0.0 0.15 0.20 0.16
B
B
B
B
Throughput 0.50 0.52 0.49 0.71 0.25
Array 1.13 1.10 1.01 1.02 Array 1.11 0.97 1.01 1.02 Array 1.92 1.24 0.47 1.38 Array 0.25 0.54 0.0 0.34
Throughput B C 0.32 0.75 0.31 0.75 0.32 0.75 0.36 0.75 Throughput B C 0.31 0.67 0.31 0.57 0.29 0.75 0.51 0.75 Throughput B C 0.27 0.97 0.24 0.70 0.34 0.24 0.58 2.43 Throughput B C 0.15 0.17 0.0 0.62 0.27 0.0 0.27 0.29
Table N.4: Mean relative error
C
C
C
C
Latency 0.38 0.46 0.43 0.55 0.29
Array 1.32 1.32 1.28 1.28 Array 1.37 1.26 1.28 1.28 Array 0.90 0.98 0.58 0.54 Array 0.28 0.29 0.19 0.0
D 0.57 0.57 0.57 0.58
A 0.32 0.32 0.32 0.33
D 0.57 0.53 0.57 0.58
A 0.32 0.32 0.27 0.29
D 0.43 1.55 0.40 0.37
A 0.21 0.24 0.31 0.33
D 0.23 0.39 0.16 0.0
A 0.0 0.14 0.24 0.32
D
D
D
D
Latency B C 0.19 0.94 0.17 0.87 0.16 0.58 0.16 0.63 Latency B C 0.14 0.97 0.17 0.64 0.15 0.58 0.18 0.61 Latency B C 0.22 2.24 0.09 0.53 0.18 0.45 0.16 0.74 Latency B C 0.16 0.27 0.0 0.47 0.28 0.0 0.26 0.43
D 0.58 0.58 0.45 0.45 D 0.73 0.45 0.45 0.45 D 0.87 0.36 0.41 0.36 D 0.30 0.31 0.25 0.0
Appendix O
Postmark model testing on Postmark samples Application Postmark (phase 1) (pmc) Postmark (phase 2) (pmt) Total used
Samples 50 50
Iters 3 3
First sample 25 25 50
Last sample 49 49
Table O.1: Multiple testing samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
181
182
Overall (Postmark) 1 0.9 0.8 Fraction
0.7 0.6 0.5
1
0.9
0.8
0.7
0.3
0.2
0
0.1
0.1
0.6
0.2
0.5
0.3
0.4
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness
0.4
Relative error Overall (Postmark)
1
1
0.9
0.9
0.8
0.8
0.7
0.7
Fraction
0.6 0.5 0.4
0.5 0.4 0.3
0.8
0.8
0.7
0.7
0.6 0.5 0.4
0.8
1
1 0.9
Fraction
1 0.9
1
Relative error Overall (Postmark)
0.9
Relative error Overall (Postmark)
0.9
0.5
0.4
0.3
0.2
0.1
0
1
0.1
0.7
Absolute Relative
0.2 0.9
0.8
0.7
0.6
0.4
0.3
0.2
0
0.1
0.1
0.5
Absolute (i=j) Absolute
0.2
0.6
0.3
0.6 0.5 0.4 0.3
Relative error
0.8
0.7
0.6
0.3
0.2
1
0.1
0.5
Relative Performance Relative Fitness
0.2 0.9
0.8
0.7
0.6
0.5
0.3
0.2
0
0.1
0.1
0.4
Relative Relative Performance
0.2
0.4
0.3
0.1
Fraction
0.6
0
Fraction
Overall (Postmark)
Relative error
Figure O.1: The cumulative distribution of relative error over all pairwise predictions.
183
APPENDIX O. POSTMARK MODEL TESTING ON POSTMARK SAMPLES
Bandwidth (Postmark) 1 0.9 0.8 Fraction
0.7 0.6 0.5
1
0.9
0.8
0.7
0.3
0.2
0
0.1
0.1
0.6
0.2
0.5
0.3
0.4
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness
0.4
Relative error Bandwidth (Postmark)
1
1
0.9
0.9
0.8
0.8
0.7
0.7
Fraction
0.6 0.5 0.4
0.5 0.4 0.3
0.8
0.8
0.7
0.7
0.6 0.5 0.4
0.8
1
1 0.9
Fraction
1 0.9
1
Relative error Bandwidth (Postmark)
0.9
Relative error Bandwidth (Postmark)
0.9
0.5
0.4
0.3
0.2
0.1
0
1
0.1
0.7
Absolute Relative
0.2 0.9
0.8
0.7
0.6
0.4
0.3
0.2
0
0.1
0.1
0.5
Absolute (i=j) Absolute
0.2
0.6
0.3
0.6 0.5 0.4 0.3
Relative error
0.8
0.7
0.6
0.3
0.2
1
0.1
0.5
Relative Performance Relative Fitness
0.2 0.9
0.8
0.7
0.6
0.5
0.3
0.2
0
0.1
0.1
0.4
Relative Relative Performance
0.2
0.4
0.3
0.1
Fraction
0.6
0
Fraction
Bandwidth (Postmark)
Relative error
Figure O.2: The cumulative distribution of relative error over all pairwise predictions.
184
Throughput (Postmark) 1 0.9 0.8 Fraction
0.7 0.6 0.5
1
0.9
0.8
0.7
0.3
0.2
0
0.1
0.1
0.6
0.2
0.5
0.3
0.4
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness
0.4
Relative error Throughput (Postmark)
1
1
0.9
0.9
0.8
0.8
0.7
0.7
Fraction
0.6 0.5 0.4
0.5 0.4 0.3
0.8
0.8
0.7
0.7
0.6 0.5 0.4
0.8
1
1 0.9
Fraction
1 0.9
1
Relative error Throughput (Postmark)
0.9
Relative error Throughput (Postmark)
0.9
0.5
0.4
0.3
0.2
0.1
0
1
0.1
0.7
Absolute Relative
0.2 0.9
0.8
0.7
0.6
0.4
0.3
0.2
0
0.1
0.1
0.5
Absolute (i=j) Absolute
0.2
0.6
0.3
0.6 0.5 0.4 0.3
Relative error
0.8
0.7
0.6
0.3
0.2
1
0.1
0.5
Relative Performance Relative Fitness
0.2 0.9
0.8
0.7
0.6
0.5
0.3
0.2
0
0.1
0.1
0.4
Relative Relative Performance
0.2
0.4
0.3
0.1
Fraction
0.6
0
Fraction
Throughput (Postmark)
Relative error
Figure O.3: The cumulative distribution of relative error over all pairwise predictions.
185
APPENDIX O. POSTMARK MODEL TESTING ON POSTMARK SAMPLES
Latency (Postmark)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error Latency (Postmark)
Latency (Postmark)
1 1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Relative Performance Relative Fitness 0
Fraction 1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative error
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.9
Relative error Latency (Postmark)
0.9
0
1
0.9
0.8
0.7
0.6
0.4
0.2
0
0.3
Relative error
Relative Relative Performance 0
Absolute Relative
Latency (Postmark) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.1
Fraction
0.1
0.1
0.5
Absolute (i=j) Absolute
0.2
0.7
0.3
0.6
0.4
0.5
0.5
0.4
0.6
0.3
0.7
Fraction
Fraction
0.8
0.2
0.9
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.1
1
Relative error
Figure O.4: The cumulative distribution of relative error over all pairwise predictions.
186
<-0
[[0 [0 [0 [1 [1 > .17 0.17 .22, .60, .99, .38, .77, =2.1 ,0 6 .22 0.60 0.99 1.38 1.77 2.16 ) ) ) ) ) )
0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
<-0
[[0 [0 [0 [1 [1 > .11 0.11 .22, .55, .88, .21, .54, =1.8 ,0 7 .22 0.55 0.88 1.21 1.54 1.87 ) ) ) ) ) )
<-0
[[[[0 [0 [0 > .19 0.19 0.11 0.03 .04, .12, .20, =0.2 , -0 , -0 , 0 8 0 0 0 .11 .03 .04 .12) .20) .28) ) ) )
<-0
[[[[[0 [0 > .44 0.44 0.30 0.17 0.03 .10, .24, =0.3 , -0 , -0 , -0 , 0 7 0 0 .30 .17 .03 .10 .24) .37) ) ) ) )
[[[[0 [0 [0 > .28 0.28 0.18 0.08 .02, .13, .23, =0.3 , -0 , -0 , 0 3 0 0 0 .18 .08 .02 .13) .23) .33) ) ) )
Fraction
<-0
[[[[0 [0 [0 > .29 0.29 0.19 0.09 .00, .10, .20, =0.3 , -0 , -0 , 0 0 0 0 0 .19 .09 .00 .10) .20) .30) ) ) )
Absolute: Latency (Avg )
Fraction
0.5 0.4 0.3 0.2 0.1 <-0
[[[0 [0 [0 [0 > .19 0.19 0.08 .04, .15, .27, .38, =0.5 , -0 , 0 0 0 0 0 0 .08 .04 .15) .27) .38) .50) ) )
0
<-0
[[0 [0 [0 [0 [1 > .20 0.20 .05, .31, .57, .83, .09, =1.3 ,0 4 .05 0.31 0.57 0.83 1.09 1.34 ) ) ) ) ) )
Relative Performance: Latency (Avg )
Fraction [[0 [0 [0 [0 [0 > .14 0.14 .05, .24, .43, .62, .81, =1.0 ,0 1 .05 0.24 0.43 0.62 0.81 1.01 ) ) ) ) ) )
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.6
0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Fitness: Latency (Avg )
Fraction
Fraction
Fraction <-0
<-0
[ [ [ [ [ [ > .10 0.10 0.46 0.83 1.19 1.55 1.92 =2.2 ,0 , , , , , .46 0.83 1.19 1.55 1.92 2.28 8 ) ) ) ) ) )
Relative Fitness: Throughput (Avg ) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative: Latency (Avg ) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
Relative: Throughput (Avg )
Fraction
0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Performance: Throughput (Avg ) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Absolute: Throughput (Avg )
Fraction
Fraction
Relative Fitness: Bandwidth (Avg ) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Performance: Bandwidth (Avg )
Fraction
Relative: Bandwidth (Avg )
Fraction
Fraction
Absolute: Bandwidth (Avg ) 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
<-0
[[[0 [0 [0 [0 > .20 0.20 0.00 .19, .39, .59, .79, =0.9 , -0 , 0 9 0 0 0 0 .00 .19 .39) .59) .79) .99) ) )
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[[0 [0 [0 > .28 0.28 0.16 0.04 .09, .21, .33, =0.4 , -0 , -0 , 0 6 0 0 0 .16 .04 .09 .21) .33) .46) ) ) )
Figure O.5: The probability distribution of relative error over all pairwise predictions.
APPENDIX O. POSTMARK MODEL TESTING ON POSTMARK SAMPLES
Bandwidth
Fraction
Fraction
Overall 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
0.6 0.5 0.4 0.3 0.2 0.1 0
, -0
80
)
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
Latency
Fraction
Fraction
Throughput 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
187
0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
Figure O.6: Probability distributions of the difference in the absolute value of the relative error value−measured value (| predictedmeasured |) between the absolute and relative fitness models, over all predictions. The x value axis represent the amount by which relative error is reduced. Negative values indicate an increase in error.
188
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 7 / 2 / 39 6 / 5 / 37 3 / 3 / 42 7 / 3 / 38 8 / 6 / 34 3 / 3 / 42 13 / 5 / 30 9 / 4 / 35 9 / 4 / 35 6 / 3 / 39 6 / 5 / 37 11 / 7 / 30 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 6 / 3 / 39 20 / 6 / 22 15 / 2 / 31 0 / 0 / 48 15 / 7 / 26 11 / 1 / 36 0 / 0 / 48 19 / 4 / 25 13 / 3 / 32 0 / 0 / 48 3 / 4 / 41 20 / 8 / 20 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 7 / 4 / 37 14 / 6 / 28 5 / 5 / 38 8 / 2 / 38 17 / 8 / 23 9 / 4 / 35 10 / 2 / 36 7 / 5 / 36 10 / 4 / 34 8 / 1 / 39 4 / 7 / 37 18 / 5 / 25 nan% nan% nan%
Bandwidth AM vs. RM RM vs. RM’ 12 / 26 / 10 16 / 16 / 16 12 / 8 / 28 9 / 14 / 25 12 / 4 / 32 0 / 0 / 48 9 / 4 / 35 12 / 3 / 33 9 / 10 / 29 13 / 9 / 26 20 / 7 / 21 0 / 0 / 48 15 / 14 / 19 10 / 23 / 15 16 / 25 / 7 15 / 11 / 22 16 / 12 / 20 7 / 10 / 31 21 / 18 / 9 18 / 11 / 19 22 / 21 / 5 7 / 18 / 23 12 / 16 / 20 12 / 13 / 23 100% 0% 0% 100% 0% 0% Throughput AM vs. RM RM vs. RM’ 16 / 12 / 20 21 / 15 / 12 12 / 23 / 13 21 / 15 / 12 10 / 24 / 14 7 / 12 / 29 8 / 19 / 21 16 / 11 / 21 17 / 19 / 12 16 / 18 / 14 8 / 20 / 20 7 / 2 / 39 0 / 6 / 42 0 / 0 / 48 11 / 22 / 15 9 / 10 / 29 9 / 22 / 17 6 / 8 / 34 0 / 0 / 48 1 / 3 / 44 10 / 8 / 30 10 / 8 / 30 16 / 28 / 4 2 / 15 / 31 100% 0% 0% 100% 0% 0% Latency AM vs. RM RM vs. RM’ 16 / 25 / 7 9 / 11 / 28 27 / 17 / 4 14 / 24 / 10 12 / 10 / 26 7 / 10 / 31 7 / 14 / 27 11 / 12 / 25 17 / 24 / 7 19 / 14 / 15 15 / 12 / 21 12 / 21 / 15 8 / 16 / 24 17 / 8 / 23 12 / 19 / 17 13 / 10 / 25 17 / 20 / 11 5 / 7 / 36 10 / 22 / 16 26 / 10 / 12 13 / 20 / 15 13 / 16 / 19 21 / 25 / 2 15 / 12 / 21 100% 0% 0% 100% 0% 0%
RM’ vs. RF 19 / 22 / 7 17 / 24 / 7 17 / 24 / 7 10 / 30 / 8 12 / 27 / 9 16 / 24 / 8 7 / 31 / 10 15 / 29 / 4 21 / 23 / 4 18 / 26 / 4 21 / 23 / 4 18 / 26 / 4 100% 0% 0%
AM vs. RF 9 / 34 / 5 16 / 22 / 10 17 / 24 / 7 10 / 28 / 10 14 / 26 / 8 19 / 22 / 7 6 / 34 / 8 10 / 33 / 5 22 / 23 / 3 20 / 21 / 7 14 / 28 / 6 15 / 22 / 11 100% 0% 0%
RM’ vs. RF 15 / 24 / 9 20 / 22 / 6 14 / 26 / 8 22 / 17 / 9 19 / 24 / 5 17 / 26 / 5 32 / 10 / 6 19 / 20 / 9 18 / 24 / 6 21 / 16 / 11 22 / 22 / 4 25 / 17 / 6 100% 0% 0%
AM vs. RF 22 / 24 / 2 19 / 26 / 3 10 / 30 / 8 21 / 18 / 9 23 / 21 / 4 15 / 30 / 3 31 / 13 / 4 20 / 23 / 5 15 / 28 / 5 20 / 17 / 11 25 / 18 / 5 20 / 23 / 5 100% 0% 0%
RM’ vs. RF 20 / 24 / 4 12 / 30 / 6 26 / 18 / 4 23 / 20 / 5 20 / 26 / 2 17 / 27 / 4 19 / 27 / 2 24 / 20 / 4 23 / 21 / 4 22 / 22 / 4 15 / 28 / 5 21 / 23 / 4 100% 0% 0%
AM vs. RF 20 / 24 / 4 17 / 27 / 4 29 / 16 / 3 20 / 21 / 7 21 / 19 / 8 15 / 30 / 3 21 / 24 / 3 21 / 26 / 1 24 / 23 / 1 21 / 24 / 3 14 / 32 / 2 20 / 26 / 2 100% 0% 0%
Table O.2: Models are compared across all pairwise predictions. For each prediction, a score of 1 is assigned to the model with the lowest accuracy. If the accuracy of two models is within 0.01, a score of 1 goes to the tie category (3rd number shown). Hypotheses are tested in turn. Specifically, idealized absolute models (i = j) are compared against in-practice absolute models (i 6= j), relative models are compared to absolute models (AM vs. RM), relative models that use performance are compared to those that do not (RM vs. RM’), and relative fitness models are compared to relative models that use performance (RM’ vs. RF). In addition, relative fitness models are compared directly to absolute models (AM vs. RF).
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.07 0.08 0.07 0.07 0.07
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.07 0.06 0.09 0.07
Relative Array A Array B Array C Array D
A 0.07 0.07 0.07 0.08
Relative Performance Array A Array B Array C Array D
A 0.04 0.10 0.07 0.09
Relative Fitness Array A Array B Array C Array D
A 0.0 0.04 0.04 0.06
Bandwidth 0.07 0.08 0.07 0.07 0.05
Pairwise Array A Array 0.06 0.09 0.06 0.08 0.08 0.11 0.07 0.08 Array A Array 0.06 0.07 0.06 0.08 0.06 0.07 0.06 0.07 Array A Array 0.03 0.07 0.07 0.04 0.07 0.07 0.07 0.07 Array A Array 0.0 0.07 0.06 0.0 0.07 0.07 0.06 0.06
Bandwidth B C 0.13 0.05 0.09 0.06 0.11 0.04 0.10 0.07 Bandwidth B C 0.06 0.07 0.09 0.06 0.06 0.04 0.06 0.07 Bandwidth B C 0.05 0.06 0.05 0.06 0.07 0.03 0.05 0.08 Bandwidth B C 0.05 0.04 0.0 0.04 0.04 0.0 0.06 0.06
D 0.06 0.06 0.08 0.06
A 0.04 0.04 0.04 0.04
D 0.06 0.06 0.09 0.06
A 0.04 0.04 0.04 0.04
D 0.06 0.06 0.09 0.05
A 0.01 0.04 0.04 0.04
D 0.04 0.07 0.06 0.0
A 0.0 0.04 0.09 0.04
B
B
B
B
Throughput 0.06 0.07 0.06 0.06 0.07
Array 0.09 0.08 0.07 0.08 Array 0.08 0.08 0.07 0.08 Array 0.08 0.08 0.03 0.07 Array 0.07 0.07 0.0 0.07
Throughput B C 0.05 0.10 0.05 0.09 0.09 0.07 0.05 0.09 Throughput B C 0.06 0.08 0.05 0.08 0.05 0.07 0.06 0.11 Throughput B C 0.06 0.10 0.04 0.10 0.06 0.02 0.06 0.07 Throughput B C 0.05 0.10 0.0 0.07 0.07 0.0 0.05 0.08
Table O.3: Median relative error
C
C
C
C
Latency 0.08 0.10 0.09 0.10 0.09
Array 0.08 0.07 0.09 0.06 Array 0.07 0.07 0.09 0.06 Array 0.06 0.07 0.09 0.04 Array 0.07 0.06 0.07 0.0
D 0.11 0.09 0.10 0.08
A 0.07 0.10 0.13 0.11
D 0.09 0.08 0.08 0.08
A 0.07 0.10 0.11 0.08
D 0.07 0.08 0.07 0.05
A 0.04 0.10 0.13 0.13
D 0.08 0.06 0.06 0.0
A 0.0 0.11 0.10 0.12
D
D
D
D
Latency B C 0.13 0.09 0.11 0.10 0.13 0.08 0.10 0.09 Latency B C 0.10 0.11 0.11 0.13 0.11 0.08 0.09 0.07 Latency B C 0.11 0.11 0.04 0.12 0.10 0.05 0.09 0.09 Latency B C 0.11 0.08 0.0 0.10 0.12 0.0 0.06 0.10
D 0.07 0.08 0.07 0.06 D 0.07 0.07 0.10 0.06 D 0.06 0.07 0.08 0.04 D 0.13 0.05 0.08 0.0
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.23 0.24 0.22 0.22 0.13
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.17 0.17 0.22 0.17
Relative Array A Array B Array C Array D
A 0.17 0.18 0.22 0.49
Relative Performance Array A Array B Array C Array D
A 0.13 0.19 0.18 0.44
Relative Fitness Array A Array B Array C Array D
A 0.0 0.12 0.06 0.13
Bandwidth 0.33 0.34 0.31 0.32 0.11
Pairwise Array A Array 0.15 0.39 0.17 0.37 0.18 0.39 0.17 0.22 Array A Array 0.15 0.27 0.16 0.37 0.15 0.24 0.24 0.21 Array A Array 0.08 0.18 0.14 0.20 0.13 0.23 0.23 0.22 Array A Array 0.0 0.11 0.13 0.0 0.10 0.15 0.12 0.10
Bandwidth B C 0.69 0.22 0.68 0.22 0.69 0.22 0.35 0.22 Bandwidth B C 0.31 0.29 0.68 0.21 0.29 0.22 0.24 0.21 Bandwidth B C 0.28 0.28 0.27 0.33 0.29 0.30 0.28 0.39 Bandwidth B C 0.09 0.08 0.0 0.14 0.13 0.0 0.11 0.13
D 0.62 0.27 0.29 0.27
A 0.08 0.08 0.08 0.08
D 0.68 0.26 0.29 0.27
A 0.08 0.07 0.07 0.08
D 0.68 0.26 0.29 0.24
A 0.04 0.07 0.07 0.07
D 0.11 0.11 0.10 0.0
A 0.0 0.12 0.11 0.09
B
B
B
B
Throughput 0.12 0.14 0.12 0.13 0.13
Array 0.19 0.19 0.17 0.20 Array 0.23 0.20 0.17 0.18 Array 0.23 0.24 0.15 0.26 Array 0.11 0.17 0.0 0.17
Throughput B C 0.10 0.16 0.10 0.14 0.14 0.15 0.10 0.16 Throughput B C 0.10 0.16 0.10 0.17 0.10 0.15 0.10 0.16 Throughput B C 0.11 0.17 0.07 0.16 0.10 0.05 0.10 0.12 Throughput B C 0.09 0.13 0.0 0.18 0.12 0.0 0.10 0.16
Table O.4: Mean relative error
C
C
C
C
Latency 0.23 0.25 0.24 0.22 0.16
Array 0.34 0.25 0.24 0.22 Array 0.34 0.23 0.23 0.22 Array 0.41 0.19 0.23 0.15 Array 0.18 0.10 0.12 0.0
D 0.23 0.22 0.23 0.16
A 0.22 0.27 0.25 0.26
D 0.16 0.13 0.13 0.16
A 0.22 0.22 0.16 0.15
D 0.28 0.13 0.14 0.10
A 0.07 0.16 0.15 0.17
D 0.23 0.11 0.10 0.0
A 0.0 0.15 0.13 0.15
D
D
D
D
Latency B C 0.36 0.18 0.34 0.22 0.34 0.13 0.22 0.23 Latency B C 0.39 0.23 0.34 0.23 0.32 0.13 0.30 0.17 Latency B C 0.16 0.26 0.26 0.23 0.30 0.09 0.28 0.26 Latency B C 0.15 0.12 0.0 0.19 0.20 0.0 0.09 0.21
D 0.17 0.26 0.20 0.25 D 0.18 0.30 0.27 0.25 D 0.25 0.17 0.26 0.11 D 0.21 0.08 0.17 0.0
Appendix P
Cello model testing on Cello samples Application (srt) Total used
Samples 78
Iters 3
First sample 39 39
Last sample 77
Table P.1: Multiple testing samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
191
192
Overall (Cello)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
0.9
1 1
0.8
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Relative Performance Relative Fitness 0
1
0.9
0.8
0.7
0.6
0.5
Relative error
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
1
0.9
0.8
0.7
0.6
0.5
Relative error Overall (Cello)
0.4
0.3
Absolute Relative
Relative error
Relative Relative Performance 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Overall (Cello)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Overall (Cello)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Overall (Cello)
Relative error
Figure P.1: The cumulative distribution of relative error over all pairwise predictions.
193
APPENDIX P. CELLO MODEL TESTING ON CELLO SAMPLES
Bandwidth (Cello)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Bandwidth (Cello)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Bandwidth (Cello) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Bandwidth (Cello)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Bandwidth (Cello)
Relative error
Figure P.2: The cumulative distribution of relative error over all pairwise predictions.
194
Throughput (Cello)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error Throughput (Cello)
Throughput (Cello)
1 1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Relative Performance Relative Fitness 0
Fraction 1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative error
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.9
Relative error Throughput (Cello)
0.9
0
1
0.9
0.8
0.7
0.6
0.4
0.2
0
0.3
Relative error
Relative Relative Performance 0
Absolute Relative
Throughput (Cello) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.1
Fraction
0.1
0.1
0.5
Absolute (i=j) Absolute
0.2
0.7
0.3
0.6
0.4
0.5
0.5
0.4
0.6
0.3
0.7
Fraction
Fraction
0.8
0.2
0.9
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.1
1
Relative error
Figure P.3: The cumulative distribution of relative error over all pairwise predictions.
195
APPENDIX P. CELLO MODEL TESTING ON CELLO SAMPLES
Latency (Cello)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
0.9
1 1
0.8
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Relative Performance Relative Fitness 0
1
0.9
0.8
0.7
0.6
0.5
Relative error
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
1
0.9
0.8
0.7
0.6
0.5
Relative error Latency (Cello)
0.4
0.3
Absolute Relative
Relative error
Relative Relative Performance 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Latency (Cello)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Latency (Cello)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Latency (Cello)
Relative error
Figure P.4: The cumulative distribution of relative error over all pairwise predictions.
196
<-0
[[[[0 [0 [0 > .28 0.28 0.15 0.02 .11, .24, .38, =0.5 , -0 , -0 , 0 1 0 0 0 .15 .02 .11 .24) .38) .51) ) ) )
0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<-0
[[[0 [0 [0 [0 > .25 0.25 0.12 .01, .14, .27, .40, =0.5 , -0 , 0 3 0 0 0 0 .12 .01 .14) .27) .40) .53) ) )
<-0
[[[0 [0 [0 [0 > .21 0.21 0.09 .02, .14, .26, .37, =0.4 , -0 , 0 9 0 0 0 0 .09 .02 .14) .26) .37) .49) ) )
<-0
[[[0 [0 [0 [0 > .14 0.14 0.05 .05, .14, .24, .34, =0.4 , -0 , 0 3 0 0 0 0 .05 .05 .14) .24) .34) .43) ) )
[-0
0.05
[0.
[ [ > .20 .11 .02 07 0.16 0.24 =0.3 , -0 , -0 , 0 , 0. , 0. , 0. 3 .11 .02 .07 16) 24) 33) ) ) )
<-0
[[0 [0 [0 [1 [1 > .12 0.12 .17, .45, .74, .03, .31, =1.6 ,0 0 .17 0.45 0.74 1.03 1.31 1.60 ) ) ) ) ) )
<-0
[[[0 [0 [0 [0 > .17 0.17 0.08 .01, .10, .19, .28, =0.3 , -0 , 0 7 0 0 0 0 .08 .01 .10) .19) .28) .37) ) )
[[[[0 [0 [0 > .20 0.20 0.11 0.03 .06, .14, .23, =0.3 , -0 , -0 , 0 1 0 0 0 .11 .03 .06 .14) .23) .31) ) ) )
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[0 [0 [0 [0 [0 > .16 0.16 .04, .24, .44, .63, .83, =1.0 ,0 3 .04 0.24 0.44 0.63 0.83 1.03 ) ) ) ) ) )
Relative Performance: Latency (Avg )
Fraction
Fraction
Relative: Latency (Avg )
<-0
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Fitness: Latency (Avg )
Fraction
.20
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Fraction
Fraction
Fraction
0.2 0.15 0.1
[-0
0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
Absolute: Latency (Avg )
0.25
[-0
[[[0 [0 [0 [0 > .20 0.20 0.08 .04, .15, .27, .39, =0.5 , -0 , 0 1 0 0 0 0 .08 .04 .15) .27) .39) .51) ) )
Relative Fitness: Throughput (Avg ) 0.3
<-0
<-0
Relative: Throughput (Avg )
Fraction
0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
Relative Performance: Throughput (Avg ) 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02
Absolute: Throughput (Avg )
Fraction
Fraction
Relative Fitness: Bandwidth (Avg ) 0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
Relative Performance: Bandwidth (Avg )
Fraction
Relative: Bandwidth (Avg )
Fraction
Fraction
Absolute: Bandwidth (Avg ) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<-0
[[0 [0 [0 [0 [1 > .15 0.15 .10, .35, .61, .86, .11, =1.3 ,0 7 .10 0.35 0.61 0.86 1.11 1.37 ) ) ) ) ) )
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[0 [0 [0 [0 > .26 0.26 0.04 .18, .40, .62, .83, =1.0 , -0 , 0 5 0 0 0 1 .04 .18 .40) .62) .83) .05) ) )
Figure P.5: The probability distribution of relative error over all pairwise predictions.
197
APPENDIX P. CELLO MODEL TESTING ON CELLO SAMPLES
Bandwidth
Fraction
Fraction
Overall 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
, -0
80
)
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
, -0
80
)
Latency
Fraction
Fraction
Throughput 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 .00 .00 ,
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
Figure P.6: Probability distributions of the difference in the absolute value of the relative error value−measured value (| predictedmeasured |) between the absolute and relative fitness models, over all predictions. The x value axis represent the amount by which relative error is reduced. Negative values indicate an increase in error.
198
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 6 / 2 / 29 7 / 11 / 19 12 / 3 / 22 0 / 0 / 37 6 / 6 / 25 12 / 4 / 21 0 / 0 / 37 3 / 3 / 31 11 / 7 / 19 0 / 0 / 37 8 / 2 / 27 6 / 7 / 24 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 5 / 4 / 28 3 / 3 / 31 0 / 4 / 33 4 / 3 / 30 2 / 3 / 32 0 / 3 / 34 5 / 3 / 29 6 / 0 / 31 1 / 3 / 33 3 / 2 / 32 8 / 4 / 25 2 / 4 / 31 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 23 / 1 / 13 13 / 1 / 23 3 / 3 / 31 8 / 5 / 24 9 / 4 / 24 5 / 6 / 26 9 / 5 / 23 17 / 0 / 20 7 / 4 / 26 12 / 3 / 22 22 / 2 / 13 17 / 1 / 19 nan% nan% nan%
Bandwidth AM vs. RM RM vs. RM’ 19 / 17 / 1 2 / 3 / 32 17 / 18 / 2 20 / 14 / 3 21 / 15 / 1 10 / 24 / 3 16 / 19 / 2 16 / 21 / 0 17 / 15 / 5 10 / 26 / 1 22 / 14 / 1 7 / 15 / 15 9 / 12 / 16 12 / 24 / 1 22 / 15 / 0 11 / 26 / 0 23 / 13 / 1 13 / 23 / 1 0 / 0 / 37 14 / 23 / 0 17 / 19 / 1 5 / 4 / 28 16 / 17 / 4 20 / 11 / 6 100% 0% 0% 100% 0% 0% Throughput AM vs. RM RM vs. RM’ 20 / 17 / 0 13 / 22 / 2 4 / 1 / 32 17 / 10 / 10 15 / 13 / 9 16 / 6 / 15 17 / 12 / 8 14 / 16 / 7 14 / 3 / 20 4 / 12 / 21 11 / 25 / 1 15 / 13 / 9 16 / 14 / 7 1 / 4 / 32 5 / 16 / 16 3 / 7 / 27 11 / 25 / 1 15 / 14 / 8 13 / 16 / 8 18 / 12 / 7 16 / 21 / 0 14 / 14 / 9 6 / 2 / 29 0 / 0 / 37 100% 0% 0% 100% 0% 0% Latency AM vs. RM RM vs. RM’ 19 / 18 / 0 0 / 0 / 37 23 / 13 / 1 0 / 0 / 37 10 / 16 / 11 16 / 17 / 4 16 / 11 / 10 14 / 9 / 14 22 / 14 / 1 17 / 7 / 13 19 / 16 / 2 7 / 16 / 14 17 / 19 / 1 0 / 0 / 37 21 / 15 / 1 14 / 17 / 6 19 / 17 / 1 17 / 14 / 6 20 / 17 / 0 14 / 22 / 1 17 / 17 / 3 14 / 12 / 11 19 / 18 / 0 0 / 0 / 37 100% 0% 0% 100% 0% 0%
RM’ vs. RF 14 / 23 / 0 14 / 22 / 1 16 / 18 / 3 12 / 23 / 2 19 / 17 / 1 13 / 21 / 3 20 / 12 / 5 14 / 20 / 3 18 / 19 / 0 15 / 20 / 2 16 / 19 / 2 17 / 18 / 2 100% 0% 0%
AM vs. RF 16 / 20 / 1 17 / 19 / 1 14 / 23 / 0 11 / 23 / 3 14 / 19 / 4 15 / 21 / 1 16 / 20 / 1 11 / 26 / 0 20 / 17 / 0 12 / 24 / 1 15 / 22 / 0 18 / 16 / 3 100% 0% 0%
RM’ vs. RF 13 / 23 / 1 15 / 18 / 4 12 / 23 / 2 15 / 20 / 2 13 / 20 / 4 18 / 16 / 3 19 / 14 / 4 14 / 18 / 5 19 / 16 / 2 15 / 21 / 1 20 / 14 / 3 25 / 10 / 2 100% 0% 0%
AM vs. RF 12 / 19 / 6 19 / 16 / 2 13 / 20 / 4 15 / 20 / 2 14 / 20 / 3 13 / 22 / 2 18 / 17 / 2 8 / 25 / 4 13 / 21 / 3 13 / 22 / 2 11 / 20 / 6 25 / 9 / 3 100% 0% 0%
RM’ vs. RF 19 / 17 / 1 10 / 26 / 1 24 / 11 / 2 22 / 9 / 6 14 / 22 / 1 23 / 14 / 0 20 / 12 / 5 21 / 13 / 3 22 / 13 / 2 13 / 19 / 5 17 / 19 / 1 18 / 18 / 1 100% 0% 0%
AM vs. RF 19 / 17 / 1 18 / 18 / 1 22 / 14 / 1 21 / 13 / 3 17 / 19 / 1 23 / 14 / 0 19 / 14 / 4 22 / 12 / 3 24 / 13 / 0 16 / 19 / 2 18 / 19 / 0 22 / 15 / 0 100% 0% 0%
Table P.2: Models are compared across all pairwise predictions. For each prediction, a score of 1 is assigned to the model with the lowest accuracy. If the accuracy of two models is within 0.01, a score of 1 goes to the tie category (3rd number shown). Hypotheses are tested in turn. Specifically, idealized absolute models (i = j) are compared against in-practice absolute models (i 6= j), relative models are compared to absolute models (AM vs. RM), relative models that use performance are compared to those that do not (RM vs. RM’), and relative fitness models are compared to relative models that use performance (RM’ vs. RF). In addition, relative fitness models are compared directly to absolute models (AM vs. RF).
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.12 0.15 0.16 0.13 0.13
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.20 0.20 0.20 0.20
Relative Array A Array B Array C Array D
A 0.20 0.13 0.16 0.20
Relative Performance Array A Array B Array C Array D
A 0.07 0.16 0.13 0.10
Relative Fitness Array A Array B Array C Array D
A 0.0 0.08 0.14 0.09
Bandwidth 0.18 0.20 0.20 0.15 0.13
Pairwise Array A Array 0.11 0.17 0.13 0.11 0.12 0.15 0.12 0.18 Array A Array 0.11 0.17 0.13 0.11 0.13 0.15 0.13 0.18 Array A Array 0.04 0.16 0.12 0.06 0.12 0.12 0.10 0.17 Array A Array 0.0 0.12 0.11 0.0 0.12 0.10 0.09 0.17
Bandwidth B C 0.21 0.14 0.18 0.17 0.18 0.17 0.22 0.15 Bandwidth B C 0.23 0.15 0.18 0.20 0.23 0.17 0.23 0.15 Bandwidth B C 0.24 0.12 0.04 0.11 0.13 0.05 0.19 0.13 Bandwidth B C 0.17 0.15 0.0 0.12 0.10 0.0 0.17 0.14
D 0.20 0.20 0.20 0.13
A 0.09 0.08 0.09 0.09
D 0.23 0.23 0.23 0.13
A 0.09 0.12 0.11 0.09
D 0.12 0.16 0.19 0.06
A 0.04 0.08 0.08 0.09
D 0.11 0.10 0.15 0.0
A 0.0 0.09 0.08 0.08
B
B
B
B
Throughput 0.10 0.10 0.11 0.09 0.09
Array 0.13 0.15 0.14 0.14 Array 0.16 0.17 0.14 0.16 Array 0.16 0.13 0.04 0.16 Array 0.11 0.11 0.0 0.20
Throughput B C 0.08 0.09 0.07 0.10 0.09 0.10 0.10 0.06 Throughput B C 0.09 0.09 0.07 0.11 0.06 0.10 0.09 0.08 Throughput B C 0.08 0.11 0.06 0.09 0.06 0.03 0.08 0.08 Throughput B C 0.06 0.08 0.0 0.06 0.06 0.0 0.08 0.15
Table P.3: Median relative error
C
C
C
C
Latency 0.11 0.17 0.19 0.17 0.20
Array 0.15 0.14 0.15 0.14 Array 0.15 0.17 0.17 0.14 Array 0.12 0.14 0.16 0.06 Array 0.11 0.13 0.18 0.0
D 0.12 0.12 0.12 0.15
A 0.11 0.11 0.12 0.11
D 0.11 0.13 0.13 0.15
A 0.11 0.14 0.13 0.13
D 0.11 0.11 0.10 0.05
A 0.03 0.12 0.13 0.11
D 0.07 0.10 0.14 0.0
A 0.0 0.14 0.15 0.11
D
D
D
D
Latency B C 0.21 0.17 0.08 0.19 0.17 0.15 0.21 0.21 Latency B C 0.23 0.26 0.08 0.20 0.20 0.15 0.22 0.26 Latency B C 0.23 0.26 0.04 0.19 0.17 0.05 0.21 0.26 Latency B C 0.25 0.12 0.0 0.21 0.21 0.0 0.25 0.26
D 0.14 0.13 0.15 0.13 D 0.13 0.15 0.15 0.13 D 0.12 0.10 0.17 0.05 D 0.14 0.19 0.20 0.0
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.18 0.23 0.24 0.22 0.21
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.22 0.22 0.22 0.22
Relative Array A Array B Array C Array D
A 0.22 0.22 0.21 0.22
Relative Performance Array A Array B Array C Array D
A 0.11 0.16 0.14 0.16
Relative Fitness Array A Array B Array C Array D
A 0.0 0.14 0.18 0.14
Bandwidth 0.21 0.23 0.25 0.20 0.18
Pairwise Array A Array 0.17 0.26 0.18 0.15 0.18 0.23 0.19 0.32 Array A Array 0.17 0.26 0.20 0.15 0.19 0.23 0.19 0.29 Array A Array 0.08 0.27 0.17 0.08 0.16 0.18 0.15 0.28 Array A Array 0.0 0.22 0.15 0.0 0.17 0.19 0.16 0.25
Bandwidth B C 0.27 0.19 0.21 0.21 0.24 0.21 0.32 0.19 Bandwidth B C 0.30 0.23 0.21 0.27 0.28 0.21 0.30 0.18 Bandwidth B C 0.34 0.20 0.07 0.13 0.19 0.07 0.30 0.21 Bandwidth B C 0.24 0.16 0.0 0.21 0.16 0.0 0.26 0.16
D 0.23 0.22 0.21 0.19
A 0.12 0.14 0.14 0.15
D 0.25 0.25 0.25 0.19
A 0.12 0.14 0.13 0.13
D 0.16 0.21 0.21 0.07
A 0.05 0.10 0.10 0.13
D 0.17 0.15 0.21 0.0
A 0.0 0.09 0.12 0.13
B
B
B
B
Throughput 0.14 0.15 0.14 0.13 0.12
Array 0.28 0.26 0.23 0.33 Array 0.32 0.33 0.23 0.31 Array 0.32 0.27 0.15 0.32 Array 0.26 0.25 0.0 0.33
Throughput B C 0.13 0.11 0.12 0.11 0.15 0.11 0.15 0.09 Throughput B C 0.13 0.11 0.12 0.13 0.11 0.11 0.13 0.11 Throughput B C 0.13 0.13 0.09 0.11 0.10 0.04 0.12 0.11 Throughput B C 0.08 0.11 0.0 0.08 0.08 0.0 0.14 0.17
Table P.4: Mean relative error
C
C
C
C
Latency 0.20 0.33 0.34 0.33 0.33
Array 0.20 0.20 0.20 0.19 Array 0.19 0.21 0.21 0.19 Array 0.16 0.18 0.19 0.08 Array 0.17 0.19 0.21 0.0
D 0.18 0.21 0.21 0.23
A 0.16 0.17 0.18 0.22
D 0.17 0.19 0.19 0.23
A 0.16 0.22 0.22 0.22
D 0.18 0.17 0.17 0.09
A 0.07 0.24 0.22 0.18
D 0.13 0.15 0.17 0.0
A 0.0 0.23 0.19 0.21
D
D
D
D
Latency B C 0.38 0.53 0.11 0.45 0.29 0.36 0.48 0.72 Latency B C 0.35 0.63 0.11 0.58 0.31 0.36 0.43 0.63 Latency B C 0.35 0.63 0.08 0.59 0.25 0.34 0.43 0.63 Latency B C 0.35 0.50 0.0 0.46 0.33 0.0 0.35 0.67
D 0.17 0.17 0.17 0.16 D 0.14 0.18 0.18 0.16 D 0.14 0.15 0.19 0.07 D 0.20 0.26 0.26 0.0
Appendix Q
TPC-C model testing on TPC-C samples Application TPC-C (tpcc) Total used
Samples 50
Iters 3
First sample 25 25
Last sample 49
Table Q.1: Multiple testing samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
201
202
Overall (TPC-C)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
0.9
1 1
0.8
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Relative Performance Relative Fitness 0
1
0.9
0.8
0.7
0.6
0.5
Relative error
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
1
0.9
0.8
0.7
0.6
0.5
Relative error Overall (TPC-C)
0.4
0.3
Absolute Relative
Relative error
Relative Relative Performance 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Overall (TPC-C)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Overall (TPC-C)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Overall (TPC-C)
Relative error
Figure Q.1: The cumulative distribution of relative error over all pairwise predictions.
203
APPENDIX Q. TPC-C MODEL TESTING ON TPC-C SAMPLES
Bandwidth (TPC-C)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error Bandwidth (TPC-C)
Bandwidth (TPC-C)
1
0.8
1
Relative error Bandwidth (TPC-C)
0.9
Relative error
0.9
0
1
0.9
0.8
0.7
0.6
0.4
0.3
0.2
0
Absolute Relative
Bandwidth (TPC-C) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1 0.9
Fraction
0.8 0.7 0.6 0.5 0.4 0.3
Relative error
0.8
0.7
0.6
0.5
0.3
0.2
0.1
0
1
0.1
0.4
Relative Performance Relative Fitness
0.2 0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
Relative Relative Performance 0.1
Fraction
0.1
0.1
0.5
Absolute (i=j) Absolute
0.2
0.7
0.3
0.6
0.4
0.5
0.5
0.4
0.6
0.3
0.7
Fraction
Fraction
0.8
0.2
0.9
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.1
1
Relative error
Figure Q.2: The cumulative distribution of relative error over all pairwise predictions.
204
Throughput (TPC-C)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.8
0.8
0.7
0.7
0.6 0.5 0.4
0.8
1
1 0.9
Fraction
1 0.9
1
Relative error Throughput (TPC-C)
0.9
Relative error Throughput (TPC-C)
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute Relative 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Absolute (i=j) Absolute 0
0.6 0.5 0.4 0.3
Relative error
0.8
0.7
0.6
0.3
0.2
0.1
1
0.1
0.5
Relative Performance Relative Fitness
0.2 0.9
0.8
0.7
0.6
0.5
0.3
0.2
0
0.1
0.1
0.4
Relative Relative Performance
0.2
0.4
0.3
0
Fraction
Throughput (TPC-C)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.1
Fraction
Throughput (TPC-C)
Relative error
Figure Q.3: The cumulative distribution of relative error over all pairwise predictions.
205
APPENDIX Q. TPC-C MODEL TESTING ON TPC-C SAMPLES
Latency (TPC-C)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1 1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.9
Relative error Latency (TPC-C)
0.9
Relative error
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Latency (TPC-C) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Latency (TPC-C)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Latency (TPC-C)
Relative error
Figure Q.4: The cumulative distribution of relative error over all pairwise predictions.
206
Relative: Bandwidth (Avg )
<-0
[[[[0 [0 [0 > .14 0.14 0.07 0.00 .07, .13, .20, =0.2 , -0 , -0 , 0 7 0 0 0 .07 .00 .07 .13) .20) .27) ) ) )
0.3 0.25 0.2 0.15 0.1 0.05 0
0.2
[[[[0 [0 [0 > .13 0.13 0.07 0.00 .06, .13, .20, =0.2 , -0 , -0 , 0 6 0 0 0 .07 .00 .06 .13) .20) .26) ) ) )
0.3 0.25 0.2 0.15 0.1 0.05 0
[[[[0 [0 [0 > .15 0.15 0.08 0.01 .06, .12, .19, =0.2 , -0 , -0 , 0 6 0 0 0 .08 .01 .06 .12) .19) .26) ) ) )
<-0
[[[[0 [0 [0 > .14 0.14 0.08 0.02 .05, .11, .17, =0.2 , -0 , -0 , 0 4 0 0 0 .08 .02 .05 .11) .17) .24) ) ) )
0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
<-0
[[[[0 [0 [0 > .14 0.14 0.07 0.01 .06, .13, .19, =0.2 , -0 , -0 , 0 6 0 0 0 .07 .01 .06 .13) .19) .26) ) ) )
Absolute: Latency (Avg ) 0.2 0.15 0.1 0.05 0
<-0
[[[[[0 [0 > .21 0.21 0.14 0.07 0.00 .07, .13, =0.2 , -0 , -0 , -0 , 0 0 0 0 .14 .07 .00 .07 .13) .20) ) ) ) )
<-0
[[[0 [0 [0 [0 > .32 0.32 0.14 .04, .22, .40, .58, =0.7 , -0 , 0 6 0 0 0 0 .14 .04 .22) .40) .58) .76) ) )
Relative Performance: Latency (Avg )
Relative Fitness: Latency (Avg )
0.25
0.25 0.2 Fraction
0.2 Fraction
Fraction
Relative: Latency (Avg )
0.15 0.1 0.05
[[[[0 [0 [0 > .29 0.29 0.16 0.03 .10, .23, .36, =0.4 , -0 , -0 , 0 9 0 0 0 .16 .03 .10 .23) .36) .49) ) ) )
0.3 0.25 0.2 0.15 0.1 0.05 0
0.25 Fraction
Fraction
Fraction
0.05
<-0
[[[0 [0 [0 [0 > .12 0.12 0.06 .00, .07, .13, .19, =0.2 , -0 , 0 5 0 0 0 0 .06 .00 .07) .13) .19) .25) ) )
Relative Fitness: Throughput (Avg )
0.2
0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
Relative: Throughput (Avg )
<-0
Relative Performance: Throughput (Avg ) 0.25
0
0
<-0
Fraction
Fraction
Fraction [[[[0 [0 [0 > .20 0.20 0.13 0.06 .01, .08, .15, =0.2 , -0 , -0 , 0 2 0 0 0 .13 .06 .01 .08) .15) .22) ) ) )
0.1
0.1
Absolute: Throughput (Avg )
<-0
0.15
0.15
0.05
Relative Fitness: Bandwidth (Avg ) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
Relative Performance: Bandwidth (Avg ) 0.25 Fraction
Fraction
Fraction
Absolute: Bandwidth (Avg ) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0
0.15 0.1 0.05
<-0
[[[[0 [0 [0 > .30 0.30 0.17 0.04 .09, .22, .36, =0.4 , -0 , -0 , 0 9 0 0 0 .17 .04 .09 .22) .36) .49) ) ) )
0
<-0
[[[0 [0 [0 [0 > .29 0.29 0.11 .07, .25, .43, .61, =0.7 , -0 , 0 8 0 0 0 0 .11 .07 .25) .43) .61) .78) ) )
Figure Q.5: The probability distribution of relative error over all pairwise predictions.
207
APPENDIX Q. TPC-C MODEL TESTING ON TPC-C SAMPLES
Bandwidth
Fraction
Fraction
Overall 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
, -0
80
)
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
, -0
80
)
Latency
Fraction
Fraction
Throughput 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 .00 .00 ,
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
Figure Q.6: Probability distributions of the difference in the absolute value of the relative error value−measured value (| predictedmeasured |) between the absolute and relative fitness models, over all predictions. The x value axis represent the amount by which relative error is reduced. Negative values indicate an increase in error.
208
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 1 / 4 / 18 1 / 0 / 22 0 / 0 / 23 5 / 1 / 17 4 / 0 / 19 0 / 0 / 23 3 / 1 / 19 1 / 3 / 19 0 / 0 / 23 4 / 3 / 16 2 / 1 / 20 1 / 2 / 20 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 2 / 6 / 15 1 / 0 / 22 0 / 0 / 23 7 / 1 / 15 5 / 0 / 18 0 / 0 / 23 5 / 1 / 17 3 / 6 / 14 0 / 0 / 23 5 / 1 / 17 6 / 4 / 13 2 / 1 / 20 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 13 / 3 / 7 7 / 1 / 15 12 / 0 / 11 2 / 0 / 21 19 / 1 / 3 14 / 1 / 8 4 / 0 / 19 16 / 4 / 3 9 / 2 / 12 4 / 0 / 19 16 / 4 / 3 6 / 3 / 14 nan% nan% nan%
Bandwidth AM vs. RM RM vs. RM’ 3 / 5 / 15 4 / 4 / 15 13 / 9 / 1 0 / 0 / 23 0 / 0 / 23 0 / 0 / 23 4 / 9 / 10 7 / 6 / 10 11 / 12 / 0 0 / 0 / 23 13 / 8 / 2 0 / 0 / 23 7 / 4 / 12 0 / 0 / 23 2 / 7 / 14 0 / 0 / 23 0 / 0 / 23 3 / 3 / 17 11 / 6 / 6 0 / 0 / 23 3 / 7 / 13 0 / 0 / 23 15 / 8 / 0 12 / 11 / 0 100% 0% 0% 100% 0% 0% Throughput AM vs. RM RM vs. RM’ 5 / 4 / 14 2 / 2 / 19 14 / 9 / 0 0 / 0 / 23 0 / 0 / 23 0 / 0 / 23 4 / 12 / 7 8/6/9 11 / 11 / 1 9 / 11 / 3 10 / 4 / 9 4 / 2 / 17 7/7/9 0 / 0 / 23 4 / 4 / 15 0 / 0 / 23 0 / 0 / 23 11 / 5 / 7 14 / 9 / 0 0 / 0 / 23 2 / 9 / 12 0 / 0 / 23 15 / 8 / 0 12 / 11 / 0 100% 0% 0% 100% 0% 0% Latency AM vs. RM RM vs. RM’ 8 / 14 / 1 10 / 10 / 3 4 / 5 / 14 0 / 0 / 23 4 / 19 / 0 0 / 0 / 23 9 / 13 / 1 2 / 5 / 16 4 / 19 / 0 0 / 0 / 23 4 / 19 / 0 1 / 7 / 15 7 / 16 / 0 0 / 0 / 23 4 / 19 / 0 0 / 0 / 23 3 / 20 / 0 15 / 8 / 0 7 / 16 / 0 16 / 7 / 0 7 / 13 / 3 0 / 0 / 23 6 / 5 / 12 0 / 0 / 23 100% 0% 0% 100% 0% 0%
RM’ vs. RF 10 / 8 / 5 9 / 12 / 2 14 / 6 / 3 8 / 14 / 1 8 / 10 / 5 13 / 8 / 2 7 / 13 / 3 10 / 8 / 5 10 / 9 / 4 14 / 9 / 0 17 / 4 / 2 11 / 9 / 3 100% 0% 0%
AM vs. RF 9 / 10 / 4 14 / 7 / 2 14 / 6 / 3 6 / 15 / 2 11 / 11 / 1 12 / 10 / 1 10 / 12 / 1 5 / 11 / 7 12 / 9 / 2 13 / 7 / 3 16 / 6 / 1 15 / 7 / 1 100% 0% 0%
RM’ vs. RF 8/9/6 8 / 11 / 4 13 / 6 / 4 6 / 14 / 3 8 / 12 / 3 12 / 8 / 3 7 / 13 / 3 11 / 10 / 2 11 / 11 / 1 11 / 10 / 2 18 / 3 / 2 13 / 9 / 1 100% 0% 0%
AM vs. RF 9 / 11 / 3 12 / 9 / 2 13 / 6 / 4 5 / 15 / 3 8 / 13 / 2 12 / 5 / 6 9 / 13 / 1 10 / 10 / 3 11 / 10 / 2 12 / 9 / 2 16 / 7 / 0 15 / 8 / 0 100% 0% 0%
RM’ vs. RF 14 / 9 / 0 10 / 13 / 0 17 / 5 / 1 13 / 8 / 2 15 / 4 / 4 14 / 9 / 0 12 / 11 / 0 18 / 5 / 0 10 / 11 / 2 14 / 8 / 1 14 / 7 / 2 6 / 16 / 1 100% 0% 0%
AM vs. RF 8 / 15 / 0 10 / 13 / 0 4 / 19 / 0 8 / 13 / 2 8 / 15 / 0 6 / 16 / 1 9 / 12 / 2 13 / 10 / 0 8 / 15 / 0 13 / 8 / 2 7 / 13 / 3 11 / 12 / 0 100% 0% 0%
Table Q.2: Models are compared across all pairwise predictions. For each prediction, a score of 1 is assigned to the model with the lowest accuracy. If the accuracy of two models is within 0.01, a score of 1 goes to the tie category (3rd number shown). Hypotheses are tested in turn. Specifically, idealized absolute models (i = j) are compared against in-practice absolute models (i 6= j), relative models are compared to absolute models (AM vs. RM), relative models that use performance are compared to those that do not (RM vs. RM’), and relative fitness models are compared to relative models that use performance (RM’ vs. RF). In addition, relative fitness models are compared directly to absolute models (AM vs. RF).
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.08 0.10 0.09 0.09 0.11
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.06 0.09 0.06 0.06
Relative Array A Array B Array C Array D
A 0.06 0.09 0.07 0.07
Relative Performance Array A Array B Array C Array D
A 0.04 0.08 0.07 0.07
Relative Fitness Array A Array B Array C Array D
A 0.0 0.07 0.06 0.12
Bandwidth 0.07 0.07 0.07 0.07 0.09
Pairwise Array A Array 0.07 0.07 0.10 0.07 0.09 0.07 0.10 0.10 Array A Array 0.07 0.05 0.09 0.07 0.09 0.05 0.09 0.06 Array A Array 0.04 0.06 0.08 0.04 0.09 0.05 0.10 0.06 Array A Array 0.0 0.07 0.07 0.0 0.07 0.07 0.14 0.13
Bandwidth B C 0.05 0.06 0.07 0.07 0.05 0.06 0.07 0.05 Bandwidth B C 0.04 0.09 0.07 0.09 0.04 0.06 0.05 0.09 Bandwidth B C 0.04 0.09 0.04 0.09 0.04 0.03 0.05 0.06 Bandwidth B C 0.06 0.10 0.0 0.08 0.04 0.0 0.12 0.11
D 0.07 0.07 0.07 0.07
A 0.07 0.10 0.07 0.09
D 0.07 0.07 0.07 0.07
A 0.07 0.09 0.09 0.10
D 0.07 0.07 0.06 0.03
A 0.06 0.08 0.09 0.10
D 0.12 0.10 0.12 0.0
A 0.0 0.05 0.06 0.12
B
B
B
B
Throughput 0.07 0.07 0.07 0.07 0.08
Array 0.09 0.16 0.08 0.07 Array 0.11 0.10 0.08 0.11 Array 0.11 0.11 0.03 0.09 Array 0.10 0.11 0.0 0.12
Throughput B C 0.05 0.05 0.07 0.08 0.05 0.05 0.07 0.05 Throughput B C 0.04 0.10 0.07 0.07 0.05 0.05 0.05 0.10 Throughput B C 0.04 0.10 0.03 0.10 0.05 0.03 0.05 0.06 Throughput B C 0.04 0.08 0.0 0.07 0.06 0.0 0.10 0.11
Table Q.3: Median relative error
C
C
C
C
Latency 0.11 0.23 0.13 0.13 0.16
Array 0.11 0.11 0.09 0.08 Array 0.08 0.10 0.08 0.08 Array 0.08 0.10 0.10 0.03 Array 0.12 0.13 0.11 0.0
D 0.06 0.06 0.06 0.06
A 0.07 0.08 0.12 0.10
D 0.06 0.07 0.06 0.06
A 0.07 0.07 0.09 0.09
D 0.06 0.06 0.07 0.03
A 0.03 0.06 0.09 0.11
D 0.09 0.09 0.07 0.0
A 0.0 0.13 0.16 0.17
D
D
D
D
Latency B C 0.28 0.17 0.08 0.46 0.18 0.11 0.18 0.12 Latency B C 0.15 0.17 0.08 0.17 0.17 0.11 0.15 0.17 Latency B C 0.13 0.17 0.04 0.17 0.17 0.04 0.15 0.17 Latency B C 0.17 0.14 0.0 0.17 0.21 0.0 0.15 0.12
D 0.35 0.36 0.25 0.12 D 0.08 0.16 0.08 0.12 D 0.08 0.15 0.10 0.03 D 0.13 0.15 0.14 0.0
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.11 0.16 0.13 0.13 0.16
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.08 0.13 0.08 0.08
Relative Array A Array B Array C Array D
A 0.08 0.13 0.10 0.11
Relative Performance Array A Array B Array C Array D
A 0.07 0.08 0.10 0.11
Relative Fitness Array A Array B Array C Array D
A 0.0 0.07 0.09 0.15
Bandwidth 0.09 0.10 0.10 0.09 0.11
Pairwise Array A Array 0.09 0.15 0.13 0.12 0.12 0.16 0.12 0.17 Array A Array 0.09 0.09 0.14 0.12 0.12 0.14 0.13 0.15 Array A Array 0.06 0.09 0.10 0.06 0.12 0.14 0.14 0.15 Array A Array 0.0 0.13 0.10 0.0 0.12 0.21 0.19 0.23
Bandwidth B C 0.07 0.10 0.08 0.11 0.07 0.09 0.09 0.08 Bandwidth B C 0.06 0.11 0.08 0.11 0.06 0.09 0.07 0.11 Bandwidth B C 0.05 0.11 0.05 0.11 0.06 0.04 0.07 0.09 Bandwidth B C 0.07 0.11 0.0 0.10 0.06 0.0 0.14 0.12
D 0.12 0.12 0.12 0.12
A 0.09 0.14 0.09 0.10
D 0.12 0.12 0.12 0.12
A 0.09 0.13 0.10 0.13
D 0.12 0.12 0.10 0.06
A 0.08 0.08 0.10 0.13
D 0.23 0.13 0.12 0.0
A 0.0 0.06 0.09 0.14
B
B
B
B
Throughput 0.10 0.10 0.10 0.09 0.11
Array 0.16 0.26 0.11 0.11 Array 0.13 0.14 0.11 0.13 Array 0.13 0.13 0.04 0.12 Array 0.13 0.18 0.0 0.13
Throughput B C 0.07 0.10 0.09 0.13 0.07 0.09 0.09 0.08 Throughput B C 0.05 0.11 0.09 0.12 0.06 0.09 0.07 0.11 Throughput B C 0.05 0.11 0.04 0.11 0.06 0.04 0.07 0.08 Throughput B C 0.06 0.09 0.0 0.09 0.08 0.0 0.13 0.12
Table Q.4: Mean relative error
C
C
C
C
Latency 0.15 0.29 0.18 0.19 0.26
Array 0.20 0.21 0.17 0.14 Array 0.12 0.15 0.12 0.14 Array 0.12 0.15 0.12 0.05 Array 0.21 0.18 0.12 0.0
D 0.12 0.12 0.12 0.12
A 0.10 0.13 0.19 0.17
D 0.12 0.11 0.12 0.12
A 0.10 0.15 0.15 0.15
D 0.12 0.13 0.11 0.05
A 0.04 0.15 0.15 0.18
D 0.22 0.13 0.08 0.0
A 0.0 0.16 0.17 0.27
D
D
D
D
Latency B C 0.31 0.27 0.17 0.53 0.33 0.14 0.32 0.16 Latency B C 0.16 0.18 0.17 0.18 0.30 0.14 0.30 0.18 Latency B C 0.18 0.18 0.08 0.18 0.30 0.06 0.30 0.18 Latency B C 0.28 0.18 0.0 0.35 0.49 0.0 0.42 0.16
D 0.36 0.38 0.29 0.18 D 0.13 0.20 0.11 0.18 D 0.13 0.20 0.15 0.04 D 0.19 0.26 0.14 0.0
Appendix R
WorkloadMix model testing on FitnessDirect samples Application Direct (raw) Total used
Samples 200
Iters 3
First sample 100 100
Last sample 199
Table R.1: Multiple testing samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
211
212
Overall (FitnessDirect)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Overall (FitnessDirect)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Overall (FitnessDirect) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Overall (FitnessDirect)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Overall (FitnessDirect)
Relative error
Figure R.1: The cumulative distribution of relative error over all pairwise predictions.
APPENDIX R. WORKLOADMIX MODEL TESTING ON FITNESSDIRECT SAMPLES
Bandwidth (FitnessDirect)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Bandwidth (FitnessDirect)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Bandwidth (FitnessDirect) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Bandwidth (FitnessDirect)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Bandwidth (FitnessDirect)
Relative error
Figure R.2: The cumulative distribution of relative error over all pairwise predictions.
213
214
Throughput (FitnessDirect)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Performance Relative Fitness 0.1
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Throughput (FitnessDirect)
0.9
Relative error Throughput (FitnessDirect)
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute Relative 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Throughput (FitnessDirect)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Throughput (FitnessDirect)
Relative error
Figure R.3: The cumulative distribution of relative error over all pairwise predictions.
APPENDIX R. WORKLOADMIX MODEL TESTING ON FITNESSDIRECT SAMPLES
Latency (FitnessDirect)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Latency (FitnessDirect)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Latency (FitnessDirect) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Latency (FitnessDirect)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Latency (FitnessDirect)
Relative error
Figure R.4: The cumulative distribution of relative error over all pairwise predictions.
215
216
Relative: Bandwidth (Avg )
0.2
Fraction
Fraction
0.25 0.15 0.1 0.05 0
<-0
[[0 [0 [0 [0 [1 > .28 0.28 .00, .28, .56, .84, .11, =1.3 ,0 9 .00 0.28 0.56 0.84 1.11 1.39 ) ) ) ) ) )
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[0 [0 [0 [0 [1 > .23 0.23 .04, .31, .59, .86, .13, =1.4 ,0 1 .04 0.31 0.59 0.86 1.13 1.41 ) ) ) ) ) )
Fraction
0.1 0.05 0
<-0
[[[0 [0 [0 [0 > .22 0.22 0.00 .22, .44, .66, .88, =1.1 , -0 , 0 0 0 0 0 1 .00 .22 .44) .66) .88) .10) ) )
<-0
[[[0 [0 [0 [0 > .34 0.34 0.14 .06, .26, .45, .65, =0.8 , -0 , 0 5 0 0 0 0 .14 .06 .26) .45) .65) .85) ) )
[[0 [0 [0 [1 [1 > .12 0.12 .23, .58, .93, .28, .63, =1.9 ,0 8 .23 0.58 0.93 1.28 1.63 1.98 ) ) ) ) ) )
Fraction
<-0
[[0 [0 [0 [0 [1 > .25 0.25 .04, .34, .64, .93, .23, =1.5 ,0 2 .04 0.34 0.64 0.93 1.23 1.52 ) ) ) ) ) )
Fraction <-0
[[[0 [0 [0 [0 > .27 0.27 0.05 .17, .39, .60, .82, =1.0 , -0 , 0 4 0 0 0 1 .05 .17 .39) .60) .82) .04) ) )
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[0 [0 [1 [1 [2 > .27 0.27 .23, .73, .24, .74, .24, =2.7 ,0 5 .23 0.73 1.24 1.74 2.24 2.75 ) ) ) ) ) )
Relative Performance: Latency (Avg )
Fraction [[0 [0 [0 [1 [1 > .36 0.36 .00, .36, .72, .07, .43, =1.7 ,0 9 .00 0.36 0.72 1.07 1.43 1.79 ) ) ) ) ) )
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Absolute: Latency (Avg )
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Fitness: Latency (Avg )
Fraction
Fraction
Fraction <-0
<-0
[[0 [0 [0 [0 [1 > .22 0.22 .05, .32, .58, .85, .12, =1.3 ,0 8 .05 0.32 0.58 0.85 1.12 1.38 ) ) ) ) ) )
Relative Fitness: Throughput (Avg ) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative: Latency (Avg ) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
Relative: Throughput (Avg )
0.2 0.15
Relative Performance: Throughput (Avg ) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Absolute: Throughput (Avg ) 0.25 Fraction
Fraction
Relative Fitness: Bandwidth (Avg ) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Performance: Bandwidth (Avg )
Fraction
Absolute: Bandwidth (Avg ) 0.3
<-0
[[[0 [0 [0 [0 > .33 0.33 0.07 .19, .45, .71, .97, =1.2 , -0 , 0 3 0 0 0 1 .07 .19 .45) .71) .97) .23) ) )
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[0 [0 [0 [0 > .31 0.31 0.08 .15, .38, .60, .83, =1.0 , -0 , 0 6 0 0 0 1 .08 .15 .38) .60) .83) .06) ) )
Figure R.5: The probability distribution of relative error over all pairwise predictions.
APPENDIX R. WORKLOADMIX MODEL TESTING ON FITNESSDIRECT SAMPLES
Bandwidth
Fraction
Fraction
Overall 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
, -0
80
)
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
, -0
80
)
Latency
Fraction
Fraction
Throughput 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
217
0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
Figure R.6: Probability distributions of the difference in the absolute value of the relative error value−measured value (| predictedmeasured |) between the absolute and relative fitness models, over all predictions. The x value axis represent the amount by which relative error is reduced. Negative values indicate an increase in error.
218
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 12 / 10 / 76 29 / 12 / 57 23 / 9 / 66 12 / 8 / 78 12 / 3 / 83 2 / 5 / 91 5 / 5 / 88 13 / 6 / 79 16 / 9 / 73 11 / 9 / 78 4 / 3 / 91 11 / 5 / 82 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 24 / 20 / 54 2 / 2 / 94 25 / 18 / 55 35 / 8 / 55 4 / 3 / 91 4 / 6 / 88 32 / 8 / 58 18 / 15 / 65 13 / 14 / 71 33 / 8 / 57 4 / 7 / 87 5 / 3 / 90 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 32 / 17 / 49 46 / 14 / 38 27 / 8 / 63 37 / 21 / 40 17 / 11 / 70 7 / 7 / 84 28 / 16 / 54 19 / 12 / 67 14 / 8 / 76 35 / 21 / 42 9 / 4 / 85 16 / 9 / 73 nan% nan% nan%
Bandwidth AM vs. RM RM vs. RM’ 33 / 36 / 29 40 / 52 / 6 33 / 51 / 14 38 / 57 / 3 44 / 36 / 18 38 / 51 / 9 35 / 34 / 29 37 / 56 / 5 43 / 39 / 16 34 / 50 / 14 26 / 37 / 35 38 / 46 / 14 35 / 28 / 35 33 / 51 / 14 35 / 51 / 12 42 / 46 / 10 35 / 44 / 19 36 / 46 / 16 37 / 31 / 30 30 / 63 / 5 41 / 40 / 17 32 / 58 / 8 34 / 43 / 21 42 / 48 / 8 100% 0% 0% 100% 0% 0% Throughput AM vs. RM RM vs. RM’ 43 / 39 / 16 36 / 49 / 13 30 / 32 / 36 83 / 15 / 0 48 / 36 / 14 37 / 52 / 9 45 / 50 / 3 40 / 53 / 5 88 / 9 / 1 15 / 83 / 0 37 / 42 / 19 52 / 39 / 7 43 / 53 / 2 45 / 46 / 7 58 / 36 / 4 40 / 50 / 8 43 / 46 / 9 48 / 47 / 3 35 / 59 / 4 30 / 63 / 5 50 / 40 / 8 32 / 58 / 8 53 / 41 / 4 36 / 56 / 6 100% 0% 0% 100% 0% 0% Latency AM vs. RM RM vs. RM’ 49 / 41 / 8 32 / 63 / 3 32 / 59 / 7 31 / 59 / 8 28 / 54 / 16 28 / 62 / 8 38 / 54 / 6 45 / 45 / 8 46 / 35 / 17 38 / 58 / 2 36 / 42 / 20 28 / 66 / 4 37 / 52 / 9 38 / 55 / 5 49 / 40 / 9 37 / 60 / 1 32 / 39 / 27 36 / 49 / 13 36 / 50 / 12 39 / 54 / 5 40 / 46 / 12 41 / 53 / 4 53 / 36 / 9 36 / 52 / 10 100% 0% 0% 100% 0% 0%
RM’ vs. RF 38 / 56 / 4 46 / 49 / 3 32 / 58 / 8 29 / 61 / 8 49 / 47 / 2 29 / 60 / 9 32 / 60 / 6 47 / 46 / 5 46 / 47 / 5 36 / 53 / 9 39 / 50 / 9 47 / 50 / 1 100% 0% 0%
AM vs. RF 31 / 63 / 4 36 / 60 / 2 28 / 68 / 2 27 / 68 / 3 43 / 51 / 4 28 / 66 / 4 32 / 60 / 6 43 / 55 / 0 47 / 48 / 3 29 / 62 / 7 23 / 72 / 3 43 / 50 / 5 100% 0% 0%
RM’ vs. RF 35 / 57 / 6 14 / 82 / 2 35 / 58 / 5 43 / 53 / 2 43 / 50 / 5 30 / 64 / 4 31 / 64 / 3 35 / 59 / 4 41 / 50 / 7 42 / 52 / 4 37 / 60 / 1 43 / 50 / 5 100% 0% 0%
AM vs. RF 33 / 63 / 2 44 / 51 / 3 36 / 57 / 5 38 / 60 / 0 44 / 50 / 4 33 / 59 / 6 25 / 70 / 3 40 / 52 / 6 42 / 52 / 4 30 / 66 / 2 34 / 58 / 6 43 / 50 / 5 100% 0% 0%
RM’ vs. RF 38 / 53 / 7 41 / 52 / 5 38 / 56 / 4 42 / 52 / 4 33 / 59 / 6 32 / 61 / 5 52 / 43 / 3 46 / 47 / 5 36 / 60 / 2 38 / 52 / 8 32 / 61 / 5 49 / 48 / 1 100% 0% 0%
AM vs. RF 26 / 68 / 4 21 / 75 / 2 21 / 75 / 2 35 / 61 / 2 30 / 65 / 3 20 / 74 / 4 36 / 59 / 3 31 / 64 / 3 29 / 68 / 1 27 / 68 / 3 26 / 69 / 3 35 / 60 / 3 100% 0% 0%
Table R.2: Models are compared across all pairwise predictions. For each prediction, a score of 1 is assigned to the model with the lowest accuracy. If the accuracy of two models is within 0.01, a score of 1 goes to the tie category (3rd number shown). Hypotheses are tested in turn. Specifically, idealized absolute models (i = j) are compared against in-practice absolute models (i 6= j), relative models are compared to absolute models (AM vs. RM), relative models that use performance are compared to those that do not (RM vs. RM’), and relative fitness models are compared to relative models that use performance (RM’ vs. RF). In addition, relative fitness models are compared directly to absolute models (AM vs. RF).
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.22 0.25 0.24 0.19 0.14
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.21 0.21 0.20 0.20
Relative Array A Array B Array C Array D
A 0.21 0.19 0.22 0.21
Relative Performance Array A Array B Array C Array D
A 0.01 0.16 0.17 0.14
Relative Fitness Array A Array B Array C Array D
A 0.0 0.10 0.10 0.12
Bandwidth 0.21 0.23 0.21 0.17 0.13
Pairwise Array A Array 0.20 0.22 0.24 0.19 0.23 0.22 0.24 0.21 Array A Array 0.20 0.22 0.21 0.19 0.21 0.21 0.23 0.20 Array A Array 0.01 0.13 0.17 0.01 0.18 0.20 0.15 0.16 Array A Array 0.0 0.12 0.13 0.0 0.12 0.17 0.12 0.10
Bandwidth B C 0.21 0.32 0.19 0.24 0.21 0.21 0.20 0.24 Bandwidth B C 0.21 0.25 0.19 0.25 0.18 0.21 0.16 0.21 Bandwidth B C 0.12 0.17 0.01 0.24 0.19 0.01 0.10 0.25 Bandwidth B C 0.12 0.20 0.0 0.21 0.16 0.0 0.08 0.16
D 0.27 0.17 0.23 0.18
A 0.17 0.25 0.21 0.25
D 0.23 0.17 0.23 0.18
A 0.17 0.20 0.20 0.23
D 0.16 0.16 0.22 0.01
A 0.01 0.16 0.20 0.15
D 0.12 0.09 0.20 0.0
A 0.0 0.14 0.11 0.12
B
B
B
B
Throughput 0.19 0.21 0.23 0.20 0.14
Array 0.33 0.27 0.24 0.27 Array 0.27 0.42 0.24 0.31 Array 0.31 0.25 0.01 0.25 Array 0.19 0.20 0.0 0.17
Throughput B C 0.19 0.22 0.15 0.23 0.19 0.22 0.16 0.23 Throughput B C 0.19 0.24 0.15 0.91 0.26 0.22 0.20 0.33 Throughput B C 0.13 0.92 0.01 0.20 0.23 0.02 0.14 0.22 Throughput B C 0.12 0.19 0.0 0.21 0.15 0.0 0.10 0.19
Table R.3: Median relative error
C
C
C
C
Latency 0.29 0.35 0.30 0.21 0.16
Array 0.28 0.24 0.25 0.25 Array 0.26 0.20 0.23 0.25 Array 0.17 0.19 0.24 0.01 Array 0.12 0.10 0.17 0.0
D 0.22 0.20 0.21 0.21
A 0.22 0.29 0.30 0.28
D 0.24 0.17 0.20 0.21
A 0.22 0.24 0.21 0.26
D 0.17 0.23 0.23 0.01
A 0.01 0.18 0.16 0.16
D 0.10 0.12 0.16 0.0
A 0.0 0.14 0.18 0.14
D
D
D
D
Latency B C 0.34 0.58 0.30 0.33 0.30 0.31 0.34 0.34 Latency B C 0.33 0.32 0.30 0.38 0.23 0.31 0.23 0.38 Latency B C 0.15 0.23 0.01 0.30 0.16 0.02 0.19 0.30 Latency B C 0.13 0.16 0.0 0.19 0.21 0.0 0.14 0.17
D 0.53 0.39 0.40 0.39 D 0.34 0.34 0.30 0.39 D 0.22 0.18 0.26 0.01 D 0.13 0.10 0.14 0.0
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.39 0.45 0.44 0.49 0.31
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.32 0.35 0.32 0.36
Relative Array A Array B Array C Array D
A 0.32 0.36 0.43 0.36
Relative Performance Array A Array B Array C Array D
A 0.02 0.36 0.28 0.25
Relative Fitness Array A Array B Array C Array D
A 0.0 0.14 0.19 0.18
Bandwidth 0.40 0.42 0.41 0.36 0.38
Pairwise Array A Array 0.28 0.34 0.35 0.30 0.30 0.34 0.34 0.30 Array A Array 0.28 0.39 0.32 0.30 0.33 0.40 0.34 0.30 Array A Array 0.02 0.33 0.34 0.02 0.27 0.34 0.26 0.21 Array A Array 0.0 0.24 0.23 0.0 0.25 0.32 0.19 0.17
Bandwidth B C 0.27 0.80 0.26 0.73 0.28 0.71 0.26 0.71 Bandwidth B C 0.24 0.66 0.26 0.68 0.31 0.71 0.29 0.64 Bandwidth B C 0.29 0.46 0.02 0.93 0.29 0.03 0.15 0.44 Bandwidth B C 0.16 0.32 0.0 2.04 0.33 0.0 0.14 0.41
D 0.37 0.32 0.33 0.33
A 0.21 0.33 0.26 0.32
D 0.35 0.29 0.30 0.33
A 0.21 0.27 0.27 0.30
D 0.26 0.31 0.30 0.01
A 0.01 0.35 0.29 0.22
D 0.18 0.15 0.30 0.0
A 0.0 0.34 0.22 0.19
B
B
B
B
Throughput 0.26 0.28 0.40 0.75 0.25
Array 0.71 0.51 0.47 0.50 Array 0.49 0.92 0.47 0.58 Array 0.69 1.83 0.03 0.42 Array 0.32 0.88 0.0 0.35
Throughput B C 0.23 0.32 0.23 0.34 0.24 0.32 0.23 0.35 Throughput B C 0.24 0.33 0.23 1.53 0.31 0.32 0.25 0.45 Throughput B C 0.23 1.22 0.01 4.18 0.39 0.03 0.17 0.39 Throughput B C 0.16 0.37 0.0 0.29 0.31 0.0 0.15 0.31
Table R.4: Mean relative error
C
C
C
C
Latency 0.52 0.63 0.50 0.35 0.30
Array 0.60 0.53 0.52 0.53 Array 0.41 0.35 0.39 0.53 Array 0.29 0.52 0.36 0.03 Array 0.26 0.17 0.35 0.0
D 0.25 0.27 0.25 0.28
A 0.30 0.36 0.33 0.35
D 0.34 0.25 0.29 0.28
A 0.30 0.32 0.30 0.36
D 0.28 1.00 0.32 0.01
A 0.02 0.32 0.24 0.31
D 0.19 0.18 0.30 0.0
A 0.0 0.22 0.34 0.19
D
D
D
D
Latency B C 0.52 1.02 0.40 0.45 0.51 0.40 0.41 0.44 Latency B C 0.70 0.50 0.40 0.54 0.58 0.40 0.35 0.66 Latency B C 0.48 0.38 0.03 0.39 0.32 0.05 0.32 0.44 Latency B C 0.42 0.26 0.0 0.31 0.31 0.0 0.23 0.33
D 1.19 1.00 0.99 0.98 D 0.56 0.51 0.59 0.98 D 0.32 0.24 0.45 0.06 D 0.41 0.17 0.46 0.0
Appendix S
WorkloadMix model testing on FitnessBuffered samples Application Buffered (sd) Total used
Samples 200
Iters 3
First sample 100 100
Last sample 199
Table S.1: Multiple testing samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
221
222
Overall (FitnessBuffered)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Overall (FitnessBuffered)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Overall (FitnessBuffered) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Overall (FitnessBuffered)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Overall (FitnessBuffered)
Relative error
Figure S.1: The cumulative distribution of relative error over all pairwise predictions.
APPENDIX S. WORKLOADMIX MODEL TESTING ON FITNESSBUFFERED SAMPLES
Bandwidth (FitnessBuffered)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Performance Relative Fitness 0.1
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Bandwidth (FitnessBuffered)
0.9
0.8
Relative error Bandwidth (FitnessBuffered)
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute Relative 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Bandwidth (FitnessBuffered)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Bandwidth (FitnessBuffered)
Relative error
Figure S.2: The cumulative distribution of relative error over all pairwise predictions.
223
224
Throughput (FitnessBuffered)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Performance Relative Fitness 0.1
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Throughput (FitnessBuffered)
0.9
0.8
Relative error Throughput (FitnessBuffered)
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute Relative 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Throughput (FitnessBuffered)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Throughput (FitnessBuffered)
Relative error
Figure S.3: The cumulative distribution of relative error over all pairwise predictions.
APPENDIX S. WORKLOADMIX MODEL TESTING ON FITNESSBUFFERED SAMPLES
Latency (FitnessBuffered)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Latency (FitnessBuffered)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Latency (FitnessBuffered) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Latency (FitnessBuffered)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Latency (FitnessBuffered)
Relative error
Figure S.4: The cumulative distribution of relative error over all pairwise predictions.
225
226
<-0
[[[0 [0 [0 [0 > .28 0.28 0.06 .16, .38, .60, .82, =1.0 , -0 , 0 4 0 0 0 1 .06 .16 .38) .60) .82) .04) ) )
0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[0 [0 [0 [0 > .32 0.32 0.15 .02, .19, .37, .54, =0.7 , -0 , 0 1 0 0 0 0 .15 .02 .19) .37) .54) .71) ) )
Fraction
0.1 0.05 0
<-0
[[[[0 [0 [0 > .28 0.28 0.14 0.00 .14, .27, .41, =0.5 , -0 , -0 , 0 5 0 0 0 .14 .00 .14 .27) .41) .55) ) ) )
<-0
[[[0 [0 [0 [0 > .26 0.26 0.07 .12, .31, .50, .68, =0.8 , -0 , 0 7 0 0 0 0 .07 .12 .31) .50) .68) .87) ) )
[[[0 [0 [0 [0 > .27 0.27 0.12 .02, .17, .31, .46, =0.6 , -0 , 0 1 0 0 0 0 .12 .02 .17) .31) .46) .61) ) )
Fraction
<-0
[[0 [0 [0 [0 [1 > .21 0.21 .04, .29, .54, .79, .04, =1.2 ,0 9 .04 0.29 0.54 0.79 1.04 1.29 ) ) ) ) ) )
Fraction <-0
[[[0 [0 [0 [0 > .24 0.24 0.06 .11, .29, .46, .63, =0.8 , -0 , 0 1 0 0 0 0 .06 .11 .29) .46) .63) .81) ) )
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[0 [0 [0 [1 > .31 0.31 0.02 .28, .57, .87, .16, =1.4 , -0 , 0 6 0 0 1 1 .02 .28 .57) .87) .16) .46) ) )
Relative Performance: Latency (Avg )
Fraction [[[0 [0 [0 [1 > .32 0.32 0.05 .21, .48, .74, .01, =1.2 , -0 , 0 7 0 0 1 1 .05 .21 .48) .74) .01) .27) ) )
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Absolute: Latency (Avg )
0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Fitness: Latency (Avg )
Fraction
Fraction
Fraction <-0
<-0
[[[0 [0 [0 [0 > .23 0.23 0.09 .06, .20, .34, .48, =0.6 , -0 , 0 2 0 0 0 0 .09 .06 .20) .34) .48) .62) ) )
Relative Fitness: Throughput (Avg ) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative: Latency (Avg ) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
Relative: Throughput (Avg )
0.2 0.15
Relative Performance: Throughput (Avg ) 0.3 0.25 0.2 0.15 0.1 0.05 0
0.3 0.25 0.2 0.15 0.1 0.05 0
Absolute: Throughput (Avg ) 0.25 Fraction
Fraction
Relative Fitness: Bandwidth (Avg ) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Performance: Bandwidth (Avg )
Fraction
Relative: Bandwidth (Avg )
Fraction
Fraction
Absolute: Bandwidth (Avg ) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[[0 [0 [0 > .35 0.35 0.18 0.01 .16, .33, .50, =0.6 , -0 , -0 , 0 6 0 0 0 .18 .01 .16 .33) .50) .66) ) ) )
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[[0 [0 [0 > .34 0.34 0.17 0.01 .16, .33, .49, =0.6 , -0 , -0 , 0 6 0 0 0 .17 .01 .16 .33) .49) .66) ) ) )
Figure S.5: The probability distribution of relative error over all pairwise predictions.
APPENDIX S. WORKLOADMIX MODEL TESTING ON FITNESSBUFFERED SAMPLES
Bandwidth
Fraction
Fraction
Overall 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
, -0
80
)
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
, -0
80
)
Latency
Fraction
Fraction
Throughput 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
227
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
Figure S.6: Probability distributions of the difference in the absolute value of the relative error value−measured value (| predictedmeasured |) between the absolute and relative fitness models, over all predictions. The x value axis represent the amount by which relative error is reduced. Negative values indicate an increase in error.
228
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 7 / 4 / 87 20 / 9 / 69 10 / 6 / 82 8 / 8 / 82 16 / 6 / 76 3 / 12 / 83 14 / 10 / 74 7 / 10 / 81 16 / 10 / 72 5 / 12 / 81 7 / 4 / 87 17 / 9 / 72 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 7 / 3 / 88 12 / 10 / 76 5 / 3 / 90 14 / 8 / 76 5 / 12 / 81 7 / 6 / 85 24 / 14 / 60 13 / 3 / 82 12 / 12 / 74 11 / 9 / 78 3 / 3 / 92 8 / 10 / 80 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 6 / 6 / 86 18 / 8 / 72 9 / 6 / 83 9 / 5 / 84 15 / 6 / 77 6 / 2 / 90 11 / 6 / 81 6 / 11 / 81 9 / 9 / 80 9 / 5 / 84 4 / 7 / 87 16 / 9 / 73 nan% nan% nan%
Bandwidth AM vs. RM RM vs. RM’ 34 / 20 / 44 43 / 49 / 6 39 / 48 / 11 36 / 47 / 15 36 / 46 / 16 24 / 52 / 22 46 / 39 / 13 34 / 59 / 5 40 / 44 / 14 37 / 56 / 5 37 / 49 / 12 26 / 62 / 10 32 / 46 / 20 45 / 41 / 12 50 / 31 / 17 39 / 50 / 9 33 / 47 / 18 33 / 54 / 11 37 / 35 / 26 32 / 59 / 7 38 / 31 / 29 30 / 61 / 7 40 / 43 / 15 37 / 52 / 9 100% 0% 0% 100% 0% 0% Throughput AM vs. RM RM vs. RM’ 34 / 45 / 19 44 / 47 / 7 38 / 54 / 6 61 / 37 / 0 31 / 51 / 16 48 / 38 / 12 43 / 48 / 7 35 / 53 / 10 63 / 29 / 6 27 / 65 / 6 38 / 46 / 14 33 / 56 / 9 45 / 45 / 8 31 / 56 / 11 46 / 47 / 5 38 / 57 / 3 41 / 46 / 11 37 / 53 / 8 42 / 45 / 11 30 / 56 / 12 43 / 39 / 16 35 / 49 / 14 36 / 31 / 31 33 / 55 / 10 100% 0% 0% 100% 0% 0% Latency AM vs. RM RM vs. RM’ 38 / 50 / 10 41 / 44 / 13 39 / 45 / 14 28 / 65 / 5 33 / 41 / 24 28 / 54 / 16 45 / 43 / 10 29 / 59 / 10 43 / 41 / 14 38 / 56 / 4 35 / 51 / 12 28 / 58 / 12 43 / 37 / 18 33 / 54 / 11 35 / 39 / 24 51 / 39 / 8 40 / 34 / 24 41 / 42 / 15 40 / 39 / 19 31 / 60 / 7 41 / 44 / 13 32 / 56 / 10 47 / 43 / 8 40 / 49 / 9 100% 0% 0% 100% 0% 0%
RM’ vs. RF 31 / 52 / 15 42 / 50 / 6 40 / 51 / 7 39 / 52 / 7 43 / 50 / 5 36 / 49 / 13 35 / 56 / 7 53 / 38 / 7 43 / 44 / 11 39 / 51 / 8 38 / 48 / 12 44 / 47 / 7 100% 0% 0%
AM vs. RF 38 / 57 / 3 40 / 52 / 6 22 / 69 / 7 32 / 60 / 6 38 / 54 / 6 30 / 63 / 5 34 / 58 / 6 52 / 39 / 7 34 / 58 / 6 33 / 60 / 5 36 / 56 / 6 41 / 51 / 6 100% 0% 0%
RM’ vs. RF 27 / 66 / 5 32 / 63 / 3 33 / 61 / 4 36 / 57 / 5 40 / 53 / 5 42 / 52 / 4 41 / 49 / 8 45 / 45 / 8 47 / 41 / 10 29 / 60 / 9 29 / 58 / 11 51 / 43 / 4 100% 0% 0%
AM vs. RF 26 / 69 / 3 32 / 58 / 8 30 / 66 / 2 15 / 79 / 4 31 / 63 / 4 28 / 66 / 4 36 / 58 / 4 43 / 50 / 5 39 / 51 / 8 23 / 71 / 4 29 / 65 / 4 40 / 52 / 6 100% 0% 0%
RM’ vs. RF 26 / 67 / 5 44 / 49 / 5 52 / 44 / 2 42 / 45 / 11 38 / 55 / 5 31 / 56 / 11 44 / 48 / 6 41 / 51 / 6 38 / 52 / 8 47 / 50 / 1 38 / 56 / 4 45 / 51 / 2 100% 0% 0%
AM vs. RF 29 / 64 / 5 30 / 63 / 5 40 / 53 / 5 31 / 62 / 5 32 / 59 / 7 25 / 68 / 5 36 / 58 / 4 40 / 51 / 7 40 / 51 / 7 26 / 65 / 7 29 / 61 / 8 31 / 64 / 3 100% 0% 0%
Table S.2: Models are compared across all pairwise predictions. For each prediction, a score of 1 is assigned to the model with the lowest accuracy. If the accuracy of two models is within 0.01, a score of 1 goes to the tie category (3rd number shown). Hypotheses are tested in turn. Specifically, idealized absolute models (i = j) are compared against in-practice absolute models (i 6= j), relative models are compared to absolute models (AM vs. RM), relative models that use performance are compared to those that do not (RM vs. RM’), and relative fitness models are compared to relative models that use performance (RM’ vs. RF). In addition, relative fitness models are compared directly to absolute models (AM vs. RF).
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.18 0.19 0.18 0.14 0.11
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.15 0.14 0.16 0.14
Relative Array A Array B Array C Array D
A 0.15 0.16 0.14 0.15
Relative Performance Array A Array B Array C Array D
A 0.01 0.11 0.15 0.11
Relative Fitness Array A Array B Array C Array D
A 0.0 0.10 0.09 0.09
Bandwidth 0.16 0.17 0.16 0.12 0.11
Pairwise Array A Array 0.18 0.15 0.18 0.15 0.19 0.16 0.17 0.16 Array A Array 0.18 0.14 0.17 0.15 0.19 0.16 0.18 0.15 Array A Array 0.01 0.13 0.12 0.01 0.15 0.15 0.12 0.10 Array A Array 0.0 0.09 0.11 0.0 0.11 0.14 0.09 0.07
Bandwidth B C 0.12 0.20 0.12 0.20 0.10 0.18 0.12 0.19 Bandwidth B C 0.14 0.18 0.12 0.19 0.15 0.18 0.17 0.20 Bandwidth B C 0.14 0.13 0.01 0.14 0.13 0.01 0.09 0.14 Bandwidth B C 0.09 0.14 0.0 0.15 0.13 0.0 0.07 0.15
D 0.20 0.17 0.23 0.17
A 0.20 0.20 0.20 0.20
D 0.17 0.19 0.18 0.17
A 0.20 0.14 0.21 0.19
D 0.11 0.10 0.13 0.01
A 0.01 0.12 0.15 0.14
D 0.10 0.10 0.12 0.0
A 0.0 0.10 0.12 0.07
B
B
B
B
Throughput 0.19 0.19 0.18 0.14 0.10
Array 0.22 0.21 0.19 0.21 Array 0.21 0.24 0.19 0.21 Array 0.20 0.16 0.01 0.17 Array 0.14 0.14 0.0 0.18
Throughput B C 0.15 0.20 0.14 0.20 0.16 0.21 0.14 0.21 Throughput B C 0.11 0.17 0.14 0.29 0.17 0.21 0.11 0.21 Throughput B C 0.11 0.26 0.01 0.15 0.13 0.01 0.11 0.19 Throughput B C 0.07 0.16 0.0 0.12 0.11 0.0 0.07 0.18
Table S.3: Median relative error
C
C
C
C
Latency 0.20 0.21 0.22 0.15 0.13
Array 0.21 0.20 0.21 0.20 Array 0.17 0.21 0.21 0.20 Array 0.13 0.12 0.16 0.01 Array 0.11 0.09 0.14 0.0
D 0.22 0.20 0.21 0.21
A 0.19 0.19 0.20 0.19
D 0.15 0.22 0.20 0.21
A 0.19 0.20 0.25 0.21
D 0.16 0.13 0.15 0.01
A 0.01 0.12 0.15 0.11
D 0.10 0.09 0.16 0.0
A 0.0 0.13 0.13 0.10
D
D
D
D
Latency B C 0.20 0.24 0.20 0.22 0.18 0.18 0.20 0.24 Latency B C 0.18 0.27 0.20 0.29 0.16 0.18 0.15 0.22 Latency B C 0.19 0.19 0.01 0.18 0.20 0.01 0.10 0.19 Latency B C 0.14 0.12 0.0 0.15 0.17 0.0 0.08 0.18
D 0.20 0.21 0.21 0.19 D 0.21 0.20 0.24 0.19 D 0.12 0.12 0.18 0.01 D 0.13 0.08 0.14 0.0
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.28 0.29 0.29 0.22 0.18
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.25 0.25 0.28 0.23
Relative Array A Array B Array C Array D
A 0.25 0.27 0.19 0.25
Relative Performance Array A Array B Array C Array D
A 0.01 0.15 0.19 0.13
Relative Fitness Array A Array B Array C Array D
A 0.0 0.12 0.14 0.12
Bandwidth 0.24 0.25 0.23 0.19 0.17
Pairwise Array A Array 0.25 0.32 0.26 0.28 0.27 0.29 0.25 0.27 Array A Array 0.25 0.24 0.28 0.28 0.27 0.26 0.28 0.28 Array A Array 0.02 0.23 0.18 0.02 0.20 0.22 0.16 0.17 Array A Array 0.0 0.13 0.15 0.0 0.19 0.23 0.14 0.13
Bandwidth B C 0.20 0.32 0.20 0.32 0.20 0.26 0.21 0.31 Bandwidth B C 0.22 0.27 0.20 0.26 0.21 0.26 0.21 0.24 Bandwidth B C 0.18 0.26 0.01 0.22 0.19 0.02 0.13 0.24 Bandwidth B C 0.12 0.23 0.0 0.19 0.23 0.0 0.11 0.21
D 0.25 0.22 0.27 0.25
A 0.27 0.29 0.27 0.27
D 0.22 0.23 0.24 0.25
A 0.27 0.30 0.34 0.32
D 0.16 0.19 0.20 0.01
A 0.02 0.22 0.22 0.19
D 0.14 0.13 0.25 0.0
A 0.0 0.15 0.22 0.14
B
B
B
B
Throughput 0.26 0.27 0.30 0.22 0.19
Array 0.35 0.33 0.30 0.34 Array 0.32 0.33 0.30 0.27 Array 0.27 0.25 0.02 0.25 Array 0.23 0.20 0.0 0.23
Throughput B C 0.23 0.27 0.21 0.26 0.25 0.27 0.21 0.28 Throughput B C 0.18 0.30 0.21 0.33 0.22 0.27 0.30 0.27 Throughput B C 0.16 0.32 0.01 0.22 0.17 0.01 0.14 0.21 Throughput B C 0.10 0.23 0.0 0.19 0.23 0.0 0.12 0.22
Table S.4: Mean relative error
C
C
C
C
Latency 0.33 0.35 0.33 0.24 0.20
Array 0.29 0.28 0.30 0.28 Array 0.27 0.27 0.36 0.28 Array 0.20 0.20 0.25 0.02 Array 0.18 0.17 0.24 0.0
D 0.32 0.32 0.31 0.30
A 0.24 0.23 0.25 0.24
D 0.29 0.29 0.44 0.30
A 0.24 0.28 0.29 0.28
D 0.25 0.24 0.27 0.01
A 0.02 0.18 0.19 0.17
D 0.17 0.23 0.26 0.0
A 0.0 0.17 0.22 0.16
D
D
D
D
Latency B C 0.52 0.45 0.44 0.40 0.41 0.36 0.38 0.42 Latency B C 0.34 0.39 0.44 0.42 0.36 0.36 0.35 0.30 Latency B C 0.35 0.24 0.03 0.32 0.30 0.03 0.25 0.30 Latency B C 0.18 0.24 0.0 0.22 0.23 0.0 0.16 0.25
D 0.30 0.30 0.33 0.28 D 0.31 0.28 0.40 0.28 D 0.18 0.16 0.26 0.04 D 0.23 0.14 0.21 0.0
Appendix T
WorkloadMix model testing on FitnessFS samples Application FS (fs) Total used
Samples 200
Iters 3
First sample 100 100
Last sample 199
Table T.1: Multiple testing samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
231
232
Overall (FitnessFS)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Overall (FitnessFS)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Overall (FitnessFS) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Overall (FitnessFS)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Overall (FitnessFS)
Relative error
Figure T.1: The cumulative distribution of relative error over all pairwise predictions.
233
APPENDIX T. WORKLOADMIX MODEL TESTING ON FITNESSFS SAMPLES
Bandwidth (FitnessFS)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Bandwidth (FitnessFS)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Bandwidth (FitnessFS) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Bandwidth (FitnessFS)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Bandwidth (FitnessFS)
Relative error
Figure T.2: The cumulative distribution of relative error over all pairwise predictions.
234
Throughput (FitnessFS)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Throughput (FitnessFS)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Throughput (FitnessFS) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Throughput (FitnessFS)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Throughput (FitnessFS)
Relative error
Figure T.3: The cumulative distribution of relative error over all pairwise predictions.
235
APPENDIX T. WORKLOADMIX MODEL TESTING ON FITNESSFS SAMPLES
Latency (FitnessFS)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Latency (FitnessFS)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Latency (FitnessFS) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Latency (FitnessFS)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Latency (FitnessFS)
Relative error
Figure T.4: The cumulative distribution of relative error over all pairwise predictions.
236
Relative: Bandwidth (Avg )
Fraction
0.25 0.2 0.15 0.1 0.05 0
<-0
[[0 [0 [0 [0 [1 > .15 0.15 .11, .36, .61, .86, .11, =1.3 ,0 6 .11 0.36 0.61 0.86 1.11 1.36 ) ) ) ) ) )
[[0 [0 [0 [0 [0 > .18 0.18 .05, .28, .50, .73, .96, =1.1 ,0 8 .05 0.28 0.50 0.73 0.96 1.18 ) ) ) ) ) )
[[[[0 [0 [0 > .20 0.20 0.10 0.01 .09, .19, .29, =0.3 , -0 , -0 , 0 9 0 0 0 .10 .01 .09 .19) .29) .39) ) ) )
<-0
[[0 [0 [0 [0 [0 > .16 0.16 .05, .25, .46, .67, .88, =1.0 ,0 9 .05 0.25 0.46 0.67 0.88 1.09 ) ) ) ) ) )
[[0 [0 [0 [0 [0 > .16 0.16 .06, .28, .49, .71, .92, =1.1 ,0 4 .06 0.28 0.49 0.71 0.92 1.14 ) ) ) ) ) )
Absolute: Latency (Avg ) 0.5 0.4 0.3 0.2 0.1
<-0
[[[0 [0 [0 [0 > .16 0.16 0.03 .10, .23, .36, .49, =0.6 , -0 , 0 2 0 0 0 0 .03 .10 .23) .36) .49) .62) ) )
0
<-0
[[0 [0 [0 [1 [1 > .28 0.28 .13, .55, .97, .39, .81, =2.2 ,0 2 .13 0.55 0.97 1.39 1.81 2.22 ) ) ) ) ) )
Relative Performance: Latency (Avg )
Fraction
Fraction
<-0
0.6 Fraction
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Fitness: Latency (Avg )
Fraction
Fraction
Fraction [[[0 [0 [0 [0 > .20 0.20 0.08 .04, .16, .29, .41, =0.5 , -0 , 0 3 0 0 0 0 .08 .04 .16) .29) .41) .53) ) )
[[[0 [0 [0 [0 > .40 0.40 0.12 .15, .43, .71, .99, =1.2 , -0 , 0 7 0 0 0 1 .12 .15 .43) .71) .99) .27) ) )
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Fitness: Throughput (Avg )
<-0
<-0
[[[0 [0 [0 [0 > .27 0.27 0.13 .01, .15, .29, .43, =0.5 , -0 , 0 7 0 0 0 0 .13 .01 .15) .29) .43) .57) ) )
Relative: Throughput (Avg )
Fraction
Fraction
Fraction <-0
Relative: Latency (Avg ) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
Absolute: Throughput (Avg ) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Performance: Throughput (Avg ) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
Relative Fitness: Bandwidth (Avg ) 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Performance: Bandwidth (Avg )
0.3 Fraction
Fraction
Absolute: Bandwidth (Avg ) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[0 [0 [0 [0 > .29 0.29 0.13 .03, .19, .35, .51, =0.6 , -0 , 0 8 0 0 0 0 .13 .03 .19) .35) .51) .68) ) )
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[[0 [0 [0 > .34 0.34 0.18 0.02 .14, .30, .46, =0.6 , -0 , -0 , 0 2 0 0 0 .18 .02 .14 .30) .46) .62) ) ) )
Figure T.5: The probability distribution of relative error over all pairwise predictions.
APPENDIX T. WORKLOADMIX MODEL TESTING ON FITNESSFS SAMPLES
Bandwidth
Fraction
Fraction
Overall 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
, -0
80
)
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
, -0
80
)
Latency
Fraction
Fraction
Throughput 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
237
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
Figure T.6: Probability distributions of the difference in the absolute value of the relative error value−measured value (| predictedmeasured |) between the absolute and relative fitness models, over all predictions. The x value axis represent the amount by which relative error is reduced. Negative values indicate an increase in error.
238
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 18 / 9 / 71 21 / 10 / 67 17 / 20 / 61 8 / 8 / 82 12 / 11 / 75 10 / 17 / 71 15 / 7 / 76 16 / 10 / 72 16 / 19 / 63 20 / 5 / 73 22 / 6 / 70 20 / 6 / 72 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 9 / 14 / 75 22 / 17 / 59 13 / 3 / 82 37 / 10 / 51 7 / 8 / 83 4 / 7 / 87 48 / 10 / 40 12 / 5 / 81 10 / 7 / 81 41 / 12 / 45 15 / 10 / 73 10 / 7 / 81 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 27 / 15 / 56 31 / 8 / 59 10 / 7 / 81 22 / 8 / 68 16 / 6 / 76 5 / 4 / 89 27 / 8 / 63 21 / 7 / 70 5 / 4 / 89 26 / 8 / 64 20 / 7 / 71 19 / 9 / 70 nan% nan% nan%
Bandwidth AM vs. RM RM vs. RM’ 32 / 34 / 32 28 / 61 / 9 53 / 38 / 7 31 / 62 / 5 46 / 28 / 24 27 / 62 / 9 35 / 45 / 18 36 / 49 / 13 54 / 32 / 12 26 / 66 / 6 50 / 32 / 16 31 / 60 / 7 34 / 40 / 24 36 / 52 / 10 49 / 37 / 12 43 / 48 / 7 36 / 38 / 24 34 / 55 / 9 32 / 37 / 29 19 / 70 / 9 38 / 44 / 16 26 / 62 / 10 45 / 42 / 11 27 / 63 / 8 100% 0% 0% 100% 0% 0% Throughput AM vs. RM RM vs. RM’ 43 / 26 / 29 30 / 54 / 14 39 / 54 / 5 64 / 34 / 0 36 / 58 / 4 44 / 50 / 4 46 / 48 / 4 25 / 64 / 9 47 / 35 / 16 35 / 52 / 11 29 / 49 / 20 41 / 55 / 2 42 / 51 / 5 26 / 56 / 16 56 / 40 / 2 28 / 67 / 3 46 / 43 / 9 46 / 49 / 3 41 / 49 / 8 38 / 48 / 12 35 / 37 / 26 33 / 54 / 11 30 / 42 / 26 45 / 50 / 3 100% 0% 0% 100% 0% 0% Latency AM vs. RM RM vs. RM’ 39 / 52 / 7 39 / 57 / 2 38 / 45 / 15 31 / 57 / 10 39 / 33 / 26 33 / 57 / 8 32 / 40 / 26 42 / 45 / 11 61 / 29 / 8 30 / 64 / 4 30 / 46 / 22 30 / 53 / 15 39 / 32 / 27 31 / 50 / 17 44 / 38 / 16 39 / 42 / 17 27 / 30 / 41 37 / 37 / 24 40 / 44 / 14 33 / 62 / 3 45 / 42 / 11 22 / 66 / 10 39 / 42 / 17 41 / 48 / 9 100% 0% 0% 100% 0% 0%
RM’ vs. RF 36 / 54 / 8 34 / 58 / 6 33 / 61 / 4 24 / 62 / 12 36 / 55 / 7 39 / 55 / 4 31 / 64 / 3 36 / 55 / 7 35 / 58 / 5 25 / 60 / 13 31 / 56 / 11 35 / 54 / 9 100% 0% 0%
AM vs. RF 26 / 69 / 3 28 / 65 / 5 31 / 65 / 2 16 / 76 / 6 35 / 59 / 4 34 / 59 / 5 19 / 77 / 2 34 / 58 / 6 32 / 58 / 8 15 / 79 / 4 17 / 71 / 10 23 / 70 / 5 100% 0% 0%
RM’ vs. RF 34 / 49 / 15 32 / 63 / 3 26 / 61 / 11 48 / 41 / 9 43 / 50 / 5 41 / 53 / 4 37 / 48 / 13 48 / 40 / 10 40 / 56 / 2 35 / 55 / 8 38 / 52 / 8 48 / 44 / 6 100% 0% 0%
AM vs. RF 37 / 51 / 10 38 / 58 / 2 26 / 70 / 2 26 / 64 / 8 39 / 49 / 10 32 / 61 / 5 27 / 68 / 3 42 / 48 / 8 29 / 62 / 7 21 / 71 / 6 32 / 59 / 7 45 / 48 / 5 100% 0% 0%
RM’ vs. RF 38 / 55 / 5 47 / 45 / 6 40 / 52 / 6 36 / 59 / 3 44 / 47 / 7 40 / 48 / 10 35 / 57 / 6 35 / 56 / 7 44 / 49 / 5 39 / 54 / 5 36 / 59 / 3 33 / 59 / 6 100% 0% 0%
AM vs. RF 32 / 62 / 4 41 / 53 / 4 34 / 58 / 6 25 / 66 / 7 45 / 49 / 4 28 / 66 / 4 25 / 63 / 10 26 / 68 / 4 39 / 54 / 5 26 / 68 / 4 28 / 64 / 6 33 / 59 / 6 100% 0% 0%
Table T.2: Models are compared across all pairwise predictions. For each prediction, a score of 1 is assigned to the model with the lowest accuracy. If the accuracy of two models is within 0.01, a score of 1 goes to the tie category (3rd number shown). Hypotheses are tested in turn. Specifically, idealized absolute models (i = j) are compared against in-practice absolute models (i 6= j), relative models are compared to absolute models (AM vs. RM), relative models that use performance are compared to those that do not (RM vs. RM’), and relative fitness models are compared to relative models that use performance (RM’ vs. RF). In addition, relative fitness models are compared directly to absolute models (AM vs. RF).
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.15 0.17 0.17 0.12 0.09
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.15 0.15 0.19 0.21
Relative Array A Array B Array C Array D
A 0.15 0.16 0.17 0.22
Relative Performance Array A Array B Array C Array D
A 0.01 0.12 0.11 0.11
Relative Fitness Array A Array B Array C Array D
A 0.0 0.06 0.07 0.07
Bandwidth 0.16 0.17 0.18 0.12 0.08
Pairwise Array A Array 0.14 0.13 0.19 0.12 0.21 0.14 0.22 0.14 Array A Array 0.14 0.13 0.16 0.12 0.18 0.17 0.17 0.16 Array A Array 0.01 0.10 0.11 0.01 0.13 0.13 0.11 0.09 Array A Array 0.0 0.07 0.08 0.0 0.08 0.09 0.08 0.06
Bandwidth B C 0.14 0.18 0.12 0.17 0.15 0.17 0.16 0.19 Bandwidth B C 0.14 0.23 0.12 0.21 0.18 0.17 0.18 0.19 Bandwidth B C 0.08 0.17 0.01 0.12 0.14 0.01 0.09 0.12 Bandwidth B C 0.06 0.09 0.0 0.12 0.09 0.0 0.06 0.10
D 0.18 0.18 0.20 0.21
A 0.13 0.15 0.20 0.20
D 0.18 0.18 0.18 0.21
A 0.13 0.15 0.15 0.12
D 0.10 0.10 0.15 0.01
A 0.01 0.07 0.12 0.11
D 0.08 0.08 0.11 0.0
A 0.0 0.07 0.07 0.08
B
B
B
B
Throughput 0.14 0.16 0.15 0.11 0.09
Array 0.17 0.16 0.14 0.16 Array 0.17 0.21 0.14 0.16 Array 0.15 0.14 0.01 0.12 Array 0.10 0.11 0.0 0.11
Throughput B C 0.10 0.13 0.10 0.13 0.12 0.14 0.12 0.15 Throughput B C 0.11 0.12 0.10 0.18 0.18 0.14 0.12 0.13 Throughput B C 0.09 0.16 0.01 0.14 0.10 0.01 0.08 0.10 Throughput B C 0.06 0.09 0.0 0.09 0.10 0.0 0.06 0.12
Table T.3: Median relative error
C
C
C
C
Latency 0.14 0.17 0.17 0.13 0.10
Array 0.18 0.18 0.19 0.18 Array 0.18 0.17 0.18 0.18 Array 0.12 0.11 0.14 0.01 Array 0.09 0.09 0.11 0.0
D 0.27 0.21 0.21 0.19
A 0.16 0.23 0.23 0.23
D 0.18 0.17 0.24 0.19
A 0.16 0.16 0.20 0.18
D 0.17 0.13 0.14 0.01
A 0.01 0.19 0.17 0.12
D 0.10 0.11 0.11 0.0
A 0.0 0.11 0.09 0.10
D
D
D
D
Latency B C 0.14 0.20 0.13 0.14 0.17 0.11 0.17 0.12 Latency B C 0.14 0.16 0.13 0.30 0.15 0.11 0.18 0.13 Latency B C 0.12 0.12 0.01 0.13 0.15 0.01 0.10 0.13 Latency B C 0.09 0.11 0.0 0.12 0.07 0.0 0.08 0.09
D 0.16 0.16 0.16 0.16 D 0.17 0.16 0.16 0.16 D 0.11 0.10 0.13 0.01 D 0.08 0.08 0.11 0.0
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.28 0.36 0.32 0.19 0.16
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.32 0.40 0.51 0.53
Relative Array A Array B Array C Array D
A 0.32 0.29 0.27 0.51
Relative Performance Array A Array B Array C Array D
A 0.01 0.17 0.19 0.13
Relative Fitness Array A Array B Array C Array D
A 0.0 0.11 0.12 0.09
Bandwidth 0.27 0.31 0.30 0.18 0.13
Pairwise Array A Array 0.24 0.56 0.32 0.28 0.38 0.28 0.37 0.50 Array A Array 0.24 0.27 0.26 0.28 0.29 0.29 0.35 0.41 Array A Array 0.01 0.17 0.19 0.01 0.20 0.20 0.15 0.21 Array A Array 0.0 0.13 0.14 0.0 0.14 0.18 0.11 0.24
Bandwidth B C 0.27 0.27 0.26 0.24 0.26 0.26 0.29 0.28 Bandwidth B C 0.25 0.31 0.26 0.29 0.28 0.26 0.23 0.40 Bandwidth B C 0.14 0.20 0.01 0.21 0.21 0.02 0.13 0.19 Bandwidth B C 0.09 0.15 0.0 0.19 0.18 0.0 0.11 0.13
D 0.23 0.21 0.24 0.25
A 0.22 0.29 0.33 0.30
D 0.29 0.26 0.25 0.25
A 0.22 0.29 0.27 0.25
D 0.17 0.20 0.20 0.01
A 0.01 0.15 0.16 0.12
D 0.11 0.12 0.17 0.0
A 0.0 0.16 0.14 0.09
B
B
B
B
Throughput 0.27 0.30 0.29 0.17 0.15
Array 0.34 0.27 0.26 0.27 Array 0.26 0.33 0.26 0.34 Array 0.21 0.23 0.02 0.19 Array 0.17 0.19 0.0 0.15
Throughput B C 0.15 0.32 0.16 0.30 0.18 0.33 0.23 0.29 Throughput B C 0.20 0.20 0.16 0.30 0.24 0.33 0.26 0.21 Throughput B C 0.14 0.23 0.01 0.22 0.14 0.01 0.12 0.16 Throughput B C 0.10 0.15 0.0 0.14 0.17 0.0 0.13 0.15
Table T.4: Mean relative error
C
C
C
C
Latency 0.30 0.48 0.36 0.22 0.20
Array 0.41 0.30 0.37 0.33 Array 0.33 0.29 0.38 0.33 Array 0.19 0.18 0.20 0.01 Array 0.13 0.16 0.18 0.0
D 0.40 0.35 0.40 0.36
A 0.19 0.26 0.29 0.27
D 0.39 0.33 0.53 0.36
A 0.19 0.21 0.34 0.29
D 0.24 0.18 0.23 0.01
A 0.01 0.25 0.24 0.20
D 0.12 0.21 0.21 0.0
A 0.0 0.16 0.17 0.14
D
D
D
D
Latency B C 1.25 0.44 0.43 0.28 0.41 0.20 0.99 0.24 Latency B C 0.37 0.27 0.43 0.40 0.35 0.20 0.74 0.41 Latency B C 0.23 0.19 0.02 0.26 0.26 0.03 0.38 0.21 Latency B C 0.19 0.22 0.0 0.25 0.20 0.0 0.49 0.16
D 0.59 0.33 0.47 0.38 D 0.30 0.27 0.37 0.38 D 0.16 0.15 0.16 0.01 D 0.15 0.15 0.17 0.0
Appendix U
WorkloadMix model testing on FitnessCache samples Application Cache (cache) Total used
Samples 70
Iters 3
First sample 35 35
Last sample 69
Table U.1: Multiple testing samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
241
242
Overall (FitnessCache)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error Overall (FitnessCache)
0.9
0.9
0.8
0.8
0.7
0.7
0.6
0.6
Fraction
0.5 0.4 0.3
0.4 0.3 0.2
Relative error
0.9
1 1
0.8 0.8
0.7
Relative Performance Relative Fitness 0.6
1
0.9
0.8
0.7
0.6
0.5
0.3
0
0.4
Relative Relative Performance
0.1
0.5
0.2
0.4
0.3
0.3
0.4
0.2
0.5
0.1
Fraction
0.6
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.9
0.7
0.5
0.4
0.3
Relative error Overall (FitnessCache)
0.7
0.2
0.2
Relative error Overall (FitnessCache) 0.8
0
0.1
0
1
0
0.9
0.1
Absolute Relative
0.1 0.9
0.8
0.7
0.6
0.4
0.3
0.2
0
0.1
0
0.5
Absolute (i=j) Absolute
0.1
0.6
0.2
Fraction
0.5
0
Fraction
Overall (FitnessCache)
Relative error
Figure U.1: The cumulative distribution of relative error over all pairwise predictions.
APPENDIX U. WORKLOADMIX MODEL TESTING ON FITNESSCACHE SAMPLES
Bandwidth (FitnessCache)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error Bandwidth (FitnessCache)
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
Fraction
0.4 0.3
0.2
Relative error
0.9
1 1
0.8 0.8
0.7
0.6
Relative Performance Relative Fitness 0.5
1
0.9
0.8
0.7
0.6
0.5
0.3
0
0.4
Relative Relative Performance
0.1
0.4
0.2
0.3
0.3
0.2
0.4
0.1
Fraction
0.5
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.9
0.7
0.5
0.4
0.3
Relative error Bandwidth (FitnessCache)
0.6
0.2
0.2
Relative error Bandwidth (FitnessCache) 0.7
0
0.1
0
1
0
0.8
0.1
Absolute Relative
0.1 0.9
0.8
0.7
0.6
0.4
0.3
0.2
0
0.1
0
0.5
Absolute (i=j) Absolute
0.1
Fraction
0.3
0.6
0.2
0.4
0
Fraction
Bandwidth (FitnessCache)
Relative error
Figure U.2: The cumulative distribution of relative error over all pairwise predictions.
243
244
Throughput (FitnessCache)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error Throughput (FitnessCache)
Throughput (FitnessCache)
1 1
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Relative Performance Relative Fitness 0
Fraction 1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Relative error
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.9
0.8
Relative error Throughput (FitnessCache)
0.9
0
1
0.9
0.8
0.7
0.6
0.4
0.3
0
0.2
Relative error
Relative Relative Performance 0
Absolute Relative
Throughput (FitnessCache) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.1
Fraction
0.1
0
0.5
Absolute (i=j) Absolute
0.1
0.7
0.2
0.6
0.3
0.5
0.4
0.4
0.5
0.3
0.6
Fraction
Fraction
0.7
0.2
0.8
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.1
0.9
Relative error
Figure U.3: The cumulative distribution of relative error over all pairwise predictions.
APPENDIX U. WORKLOADMIX MODEL TESTING ON FITNESSCACHE SAMPLES
Latency (FitnessCache)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error Latency (FitnessCache)
0.9
0.9
0.8
0.8
0.7
0.7
0.6
0.6
Fraction
0.5 0.4 0.3
0.4 0.3 0.2
Relative error
0.9
1 1
0.8 0.8
0.7
Relative Performance Relative Fitness 0.6
1
0.9
0.8
0.7
0.6
0.5
0.3
0
0.4
Relative Relative Performance
0.1
0.5
0.2
0.4
0.3
0.3
0.4
0.2
0.5
0.1
Fraction
0.6
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.9
0.7
0.5
0.4
0.3
Relative error Latency (FitnessCache)
0.7
0.2
0.2
Relative error Latency (FitnessCache) 0.8
0
0.1
0
1
0
0.9
0.1
Absolute Relative
0.1 0.9
0.8
0.7
0.6
0.4
0.3
0.2
0
0.1
0
0.5
Absolute (i=j) Absolute
0.1
0.6
0.2
Fraction
0.5
0
Fraction
Latency (FitnessCache)
Relative error
Figure U.4: The cumulative distribution of relative error over all pairwise predictions.
245
246
Relative: Bandwidth (Avg ) 0.5 Fraction
0.4 0.3 0.2 0.1
<-0
[[0 [1 [2 [3 [3 > .02 0.02 .75, .51, .27, .03, .80, =4.5 ,0 6 .75 1.51 2.27 3.03 3.80 4.56 ) ) ) ) ) )
0
[ [ [ > [ [ [ .29 0.29 1.30 2.30 3.31 4.32 5.32 =6.3 , , , ,1 , , .30 2.30 3.31 4.32 5.32 6.33 3 ) ) ) ) ) )
<0
<-0
[[[0 [0 [0 [0 > .30 0.30 0.07 .15, .37, .60, .82, =1.0 , -0 , 0 4 0 0 0 1 .07 .15 .37) .60) .82) .04) ) )
<-0
[[[0 [0 [0 [0 > .28 0.28 0.03 .22, .46, .71, .96, =1.2 , -0 , 0 1 0 0 0 1 .03 .22 .46) .71) .96) .21) ) )
[0.
[ [ [ [ > .35 08 0.52 0.95 1.38 1.81 =2.2 , 0 , 0. , 0. , 1. , 1. , 2. 5 95 38 81 25 .08 52 ) ) ) ) ) )
<0
[ [ [ [ [ [ > .54 0.54 1.60 2.67 3.73 4.80 5.87 =6.9 ,1 , , , , , .60 2.67 3.73 4.80 5.87 6.93 3 ) ) ) ) ) )
<-0
[[[0 [0 [0 [1 > .34 0.34 0.04 .25, .54, .83, .13, =1.4 , -0 , 0 2 0 0 1 1 .04 .25 .54) .83) .13) .42) ) )
Absolute: Latency (Avg ) 0.5 0.4 0.3 0.2 0.1
<-0
[[[0 [0 [0 [0 > .35 0.35 0.14 .08, .29, .51, .72, =0.9 , -0 , 0 3 0 0 0 0 .14 .08 .29) .51) .72) .93) ) )
0
<-0
[[0 [0 [1 [1 [2 > .12 0.12 .33, .78, .23, .68, .13, =2.5 ,0 8 .33 0.78 1.23 1.68 2.13 2.58 ) ) ) ) ) )
Relative Performance: Latency (Avg )
Fraction
Fraction
Relative: Latency (Avg ) 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.3 0.25 0.2 0.15 0.1 0.05 0
0.6 Fraction
0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Fitness: Latency (Avg ) 0.6 0.5 Fraction
Fraction
.35
[-0
[ [ [ [ [ [ > .48 0.48 1.48 2.49 3.49 4.50 5.51 =6.5 ,1 , , , , , .48 2.49 3.49 4.50 5.51 6.51 1 ) ) ) ) ) )
Relative Fitness: Throughput (Avg )
Fraction <-0
<0
Relative: Throughput (Avg )
Fraction
0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
Relative Performance: Throughput (Avg ) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Absolute: Throughput (Avg )
Fraction
Fraction
Relative Fitness: Bandwidth (Avg ) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Performance: Bandwidth (Avg )
0.6
Fraction
Fraction
Absolute: Bandwidth (Avg ) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.4 0.3 0.2 0.1
<-0
.00
[-0
[0.
[ [ [ [ > .00 63 1.26 1.90 2.53 3.16 =3.7 , 0 , 1. , 1. , 2. , 3. , 3. 9 90 53 16 79 .63 26 ) ) ) ) ) )
0
<-0
[[0 [0 [0 [1 [1 > .21 0.21 .11, .42, .74, .05, .37, =1.6 ,0 8 .11 0.42 0.74 1.05 1.37 1.68 ) ) ) ) ) )
Figure U.5: The probability distribution of relative error over all pairwise predictions.
APPENDIX U. WORKLOADMIX MODEL TESTING ON FITNESSCACHE SAMPLES
Bandwidth
0.25
0.2
0.2
0.15
Fraction
Fraction
Overall
0.15 0.1 0.05 0
<-1 [-1 .00 .00 ,
[-0
[-0
.60
-0.
.20
80
, -0
)
0.1 0.05 0
[0. [0. >= 20 60 1.0 ,0 ,0 0 .40 .80 .00 ) ) )
,0
.40
)
Throughput
Fraction
Fraction
0.2 0.15 0.1 0.05 <-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
, -0
80
)
Latency
0.25
0
247
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
Figure U.6: Probability distributions of the difference in the absolute value of the relative error value−measured value (| predictedmeasured |) between the absolute and relative fitness models, over all predictions. The x value axis represent the amount by which relative error is reduced. Negative values indicate an increase in error.
248
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 0 / 0 / 33 0 / 0 / 33 1 / 0 / 32 1 / 0 / 32 0 / 0 / 33 1 / 1 / 31 1 / 0 / 32 2 / 0 / 31 1 / 2 / 30 1 / 0 / 32 1 / 0 / 32 0 / 0 / 33 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 2 / 1 / 30 0 / 1 / 32 0 / 0 / 33 0 / 0 / 33 0 / 0 / 33 0 / 0 / 33 2 / 0 / 31 3 / 3 / 27 0 / 0 / 33 3 / 0 / 30 2 / 2 / 29 0 / 0 / 33 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 0 / 1 / 32 1 / 0 / 32 1 / 0 / 32 0 / 1 / 32 1 / 0 / 32 1 / 0 / 32 1 / 1 / 31 0 / 4 / 29 0 / 0 / 33 1 / 1 / 31 1 / 5 / 27 0 / 0 / 33 nan% nan% nan%
Bandwidth AM vs. RM RM vs. RM’ 17 / 7 / 9 12 / 12 / 9 9 / 18 / 6 26 / 5 / 2 14 / 8 / 11 10 / 20 / 3 8 / 15 / 10 11 / 22 / 0 9 / 12 / 12 11 / 10 / 12 11 / 10 / 12 10 / 13 / 10 7 / 14 / 12 15 / 18 / 0 11 / 19 / 3 13 / 18 / 2 9 / 6 / 18 12 / 19 / 2 12 / 12 / 9 6 / 27 / 0 9 / 6 / 18 15 / 16 / 2 4 / 10 / 19 16 / 13 / 4 100% 0% 0% 100% 0% 0% Throughput AM vs. RM RM vs. RM’ 9 / 18 / 6 11 / 18 / 4 7 / 17 / 9 24 / 9 / 0 3 / 14 / 16 7 / 16 / 10 21 / 11 / 1 9 / 17 / 7 19 / 14 / 0 16 / 17 / 0 9 / 19 / 5 17 / 15 / 1 12 / 17 / 4 15 / 11 / 7 22 / 9 / 2 14 / 19 / 0 3 / 14 / 16 14 / 16 / 3 14 / 15 / 4 7 / 24 / 2 16 / 13 / 4 13 / 18 / 2 10 / 13 / 10 12 / 15 / 6 100% 0% 0% 100% 0% 0% Latency AM vs. RM RM vs. RM’ 12 / 10 / 11 16 / 17 / 0 17 / 14 / 2 10 / 20 / 3 9 / 14 / 10 7 / 9 / 17 8 / 9 / 16 16 / 13 / 4 26 / 6 / 1 6 / 27 / 0 10 / 11 / 12 19 / 13 / 1 6 / 10 / 17 22 / 8 / 3 9 / 7 / 17 13 / 10 / 10 3 / 3 / 27 5 / 15 / 13 11 / 7 / 15 8 / 17 / 8 12 / 10 / 11 18 / 15 / 0 14 / 11 / 8 12 / 21 / 0 100% 0% 0% 100% 0% 0%
RM’ vs. RF 9 / 24 / 0 5 / 28 / 0 11 / 21 / 1 8 / 24 / 1 12 / 19 / 2 9 / 23 / 1 8 / 25 / 0 8 / 25 / 0 11 / 18 / 4 5 / 27 / 1 4 / 28 / 1 8 / 21 / 4 100% 0% 0%
AM vs. RF 7 / 26 / 0 12 / 20 / 1 9 / 23 / 1 5 / 28 / 0 9 / 23 / 1 8 / 24 / 1 7 / 26 / 0 11 / 21 / 1 4 / 28 / 1 3 / 29 / 1 3 / 29 / 1 8 / 22 / 3 100% 0% 0%
RM’ vs. RF 10 / 19 / 4 10 / 23 / 0 17 / 14 / 2 18 / 10 / 5 12 / 20 / 1 10 / 20 / 3 11 / 20 / 2 21 / 11 / 1 9 / 22 / 2 15 / 15 / 3 12 / 17 / 4 13 / 18 / 2 100% 0% 0%
AM vs. RF 10 / 22 / 1 8 / 24 / 1 9 / 24 / 0 16 / 17 / 0 13 / 19 / 1 7 / 26 / 0 10 / 21 / 2 17 / 13 / 3 7 / 24 / 2 13 / 19 / 1 7 / 24 / 2 13 / 17 / 3 100% 0% 0%
RM’ vs. RF 9 / 24 / 0 12 / 20 / 1 13 / 14 / 6 12 / 21 / 0 4 / 29 / 0 8 / 22 / 3 11 / 22 / 0 11 / 19 / 3 18 / 13 / 2 15 / 16 / 2 13 / 14 / 6 7 / 25 / 1 100% 0% 0%
AM vs. RF 11 / 17 / 5 9 / 23 / 1 13 / 19 / 1 15 / 16 / 2 8 / 25 / 0 9 / 20 / 4 15 / 17 / 1 14 / 18 / 1 15 / 16 / 2 13 / 18 / 2 19 / 14 / 0 13 / 18 / 2 100% 0% 0%
Table U.2: Models are compared across all pairwise predictions. For each prediction, a score of 1 is assigned to the model with the lowest accuracy. If the accuracy of two models is within 0.01, a score of 1 goes to the tie category (3rd number shown). Hypotheses are tested in turn. Specifically, idealized absolute models (i = j) are compared against in-practice absolute models (i 6= j), relative models are compared to absolute models (AM vs. RM), relative models that use performance are compared to those that do not (RM vs. RM’), and relative fitness models are compared to relative models that use performance (RM’ vs. RF). In addition, relative fitness models are compared directly to absolute models (AM vs. RF).
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.31 0.31 0.31 0.25 0.12
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.55 0.60 0.60 0.60
Relative Array A Array B Array C Array D
A 0.55 0.43 0.46 0.51
Relative Performance Array A Array B Array C Array D
A 0.03 0.34 0.51 0.33
Relative Fitness Array A Array B Array C Array D
A 0.0 0.08 0.21 0.06
Bandwidth 0.46 0.48 0.45 0.41 0.12
Pairwise Array A Array 0.31 0.23 0.29 0.23 0.32 0.23 0.33 0.23 Array A Array 0.31 0.18 0.33 0.23 0.26 0.28 0.33 0.20 Array A Array 0.02 0.25 0.21 0.02 0.34 0.25 0.15 0.21 Array A Array 0.0 0.08 0.11 0.0 0.13 0.17 0.09 0.08
Bandwidth B C 0.47 0.39 0.47 0.39 0.50 0.39 0.47 0.39 Bandwidth B C 0.52 0.21 0.47 0.29 0.48 0.39 0.50 0.25 Bandwidth B C 0.38 0.63 0.03 0.23 0.46 0.10 0.39 0.36 Bandwidth B C 0.08 0.17 0.0 0.16 0.20 0.0 0.04 0.11
D 0.42 0.42 0.42 0.37
A 0.14 0.14 0.24 0.26
D 0.39 0.45 0.38 0.37
A 0.14 0.27 0.15 0.23
D 0.27 0.25 0.18 0.03
A 0.01 0.09 0.19 0.05
D 0.15 0.08 0.13 0.0
A 0.0 0.10 0.10 0.11
B
B
B
B
Throughput 0.31 0.33 0.28 0.19 0.14
Array 0.39 0.39 0.39 0.39 Array 0.31 0.53 0.39 0.42 Array 0.47 0.41 0.05 0.36 Array 0.18 0.17 0.0 0.22
Throughput B C 0.25 0.39 0.19 0.39 0.29 0.39 0.23 0.39 Throughput B C 0.10 0.26 0.19 0.43 0.41 0.39 0.16 0.26 Throughput B C 0.09 0.47 0.01 0.33 0.20 0.02 0.12 0.16 Throughput B C 0.10 0.17 0.0 0.17 0.21 0.0 0.05 0.25
Table U.3: Median relative error
C
C
C
C
Latency 0.16 0.16 0.19 0.19 0.12
Array 0.27 0.27 0.27 0.27 Array 0.29 0.25 0.30 0.27 Array 0.17 0.18 0.14 0.02 Array 0.12 0.08 0.10 0.0
D 0.37 0.37 0.37 0.37
A 0.16 0.13 0.16 0.16
D 0.28 0.29 0.32 0.37
A 0.16 0.25 0.19 0.16
D 0.14 0.18 0.19 0.01
A 0.02 0.21 0.32 0.12
D 0.11 0.14 0.09 0.0
A 0.0 0.11 0.12 0.10
D
D
D
D
Latency B C 0.11 0.35 0.13 0.35 0.13 0.35 0.13 0.35 Latency B C 0.12 0.66 0.13 1.57 0.11 0.35 0.10 0.53 Latency B C 0.21 0.32 0.02 0.67 0.13 0.03 0.12 0.46 Latency B C 0.07 0.17 0.0 0.12 0.11 0.0 0.12 0.32
D 0.09 0.09 0.09 0.09 D 0.05 0.07 0.09 0.09 D 0.06 0.11 0.05 0.01 D 0.09 0.04 0.10 0.0
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.63 0.75 0.84 1.06 1.13
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 1.18 1.19 1.19 1.18
Relative Array A Array B Array C Array D
A 1.18 2.05 1.28 1.71
Relative Performance Array A Array B Array C Array D
A 0.28 0.88 1.21 0.91
Relative Fitness Array A Array B Array C Array D
A 0.0 0.19 1.02 0.15
Bandwidth 1.02 1.02 1.25 1.21 1.21
Pairwise Array A Array 0.72 0.52 0.70 0.53 0.71 0.54 0.71 0.52 Array A Array 0.72 0.49 0.97 0.53 0.64 0.58 0.87 0.57 Array A Array 0.13 0.55 0.49 0.12 0.68 2.73 0.46 0.53 Array A Array 0.0 0.18 0.34 0.0 0.49 7.77 0.21 0.18
Bandwidth B C 1.02 0.73 1.02 0.73 1.12 0.73 1.05 0.73 Bandwidth B C 1.09 0.80 1.02 0.86 1.13 0.73 1.04 0.71 Bandwidth B C 1.08 2.07 0.26 0.81 0.71 0.34 1.14 0.81 Bandwidth B C 0.20 0.28 0.0 0.27 11.41 0.0 0.13 0.28
D 1.18 1.12 1.01 1.14
A 0.35 0.35 0.37 0.38
D 1.54 1.68 1.14 1.14
A 0.35 0.47 0.20 0.36
D 1.69 1.56 1.66 0.69
A 0.03 0.19 0.25 0.14
D 0.27 0.18 0.15 0.0
A 0.0 0.25 0.17 0.15
B
B
B
B
Throughput 0.44 0.44 0.39 1.38 1.86
Array 1.29 1.29 0.59 0.59 Array 0.73 1.97 0.59 0.77 Array 1.08 1.46 0.19 0.59 Array 0.41 0.28 0.0 0.35
C
C
C
C
Throughput B C 0.38 0.41 0.40 0.41 0.37 0.41 0.37 0.41 Throughput B C 0.23 0.34 0.40 0.50 0.45 0.41 0.48 0.58 Throughput B C 0.32 0.63 0.04 0.63 7.28 0.09 0.21 0.41 Throughput B C 0.19 0.26 0.0 0.27 11.75 0.0 0.17 0.37
Table U.4: Mean relative error
Latency 0.42 0.80 0.88 0.59 0.31
Array 0.77 0.75 0.62 0.66 Array 0.93 0.93 0.62 0.66 Array 0.81 0.96 2.38 0.29 Array 0.25 0.20 2.88 0.0
D
D
D
D
D 0.59 0.59 0.59 0.59
A 0.63 0.56 0.58 0.58
D 0.40 0.30 0.39 0.59
A 0.63 0.38 0.44 0.56
D 0.26 0.94 5.30 0.06
A 0.09 0.41 0.60 0.32
D 0.20 0.20 8.34 0.0
A 0.0 0.58 0.28 0.34
Latency B C 0.17 2.73 0.17 2.73 0.14 0.62 0.14 0.62 Latency B C 0.16 1.06 0.17 4.57 0.17 0.62 0.19 1.01 Latency B C 0.24 0.55 0.05 2.94 0.20 0.16 0.24 0.54 Latency B C 0.16 0.67 0.0 0.29 0.15 0.0 0.24 0.39
D 0.55 0.55 0.25 0.25 D 0.84 0.82 0.32 0.25 D 0.48 0.38 0.20 0.11 D 0.27 0.22 0.15 0.0
Appendix V
WorkloadMix model testing on Postmark samples Application Postmark (phase 1) (pmc) Postmark (phase 2) (pmt) Total used
Samples 50 50
Iters 3 3
First sample 25 25 50
Last sample 49 49
Table V.1: Multiple testing samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
251
252
Overall (Postmark) 1 0.9 0.8 Fraction
0.7 0.6 0.5
1
0.9
0.8
0.7
0.3
0.2
0
0.1
0.1
0.6
0.2
0.5
0.3
0.4
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness
0.4
Relative error Overall (Postmark)
1
1
0.9
0.9
0.8
0.8
0.7
0.7
Fraction
0.6 0.5 0.4
0.5 0.4 0.3
0.8
0.8
0.7
0.7
0.6 0.5 0.4
0.8
1
1 0.9
Fraction
1 0.9
1
Relative error Overall (Postmark)
0.9
Relative error Overall (Postmark)
0.9
0.5
0.4
0.3
0.2
0.1
0
1
0.1
0.7
Absolute Relative
0.2 0.9
0.8
0.7
0.6
0.4
0.3
0.2
0
0.1
0.1
0.5
Absolute (i=j) Absolute
0.2
0.6
0.3
0.6 0.5 0.4 0.3
Relative error
0.8
0.7
0.6
0.3
0.2
1
0.1
0.5
Relative Performance Relative Fitness
0.2 0.9
0.8
0.7
0.6
0.5
0.3
0.2
0
0.1
0.1
0.4
Relative Relative Performance
0.2
0.4
0.3
0.1
Fraction
0.6
0
Fraction
Overall (Postmark)
Relative error
Figure V.1: The cumulative distribution of relative error over all pairwise predictions.
253
APPENDIX V. WORKLOADMIX MODEL TESTING ON POSTMARK SAMPLES
Bandwidth (Postmark) 1 0.9 0.8 Fraction
0.7 0.6 0.5
1
0.9
0.8
0.7
0.3
0.2
0
0.1
0.1
0.6
0.2
0.5
0.3
0.4
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness
0.4
Relative error Bandwidth (Postmark)
1
1
0.9
0.9
0.8
0.8
0.7
0.7
Fraction
0.6 0.5 0.4
0.5 0.4 0.3
0.8
0.8
0.7
0.7
0.6 0.5 0.4
0.8
1
1 0.9
Fraction
1 0.9
1
Relative error Bandwidth (Postmark)
0.9
Relative error Bandwidth (Postmark)
0.9
0.5
0.4
0.3
0.2
0.1
0
1
0.1
0.7
Absolute Relative
0.2 0.9
0.8
0.7
0.6
0.4
0.3
0.2
0
0.1
0.1
0.5
Absolute (i=j) Absolute
0.2
0.6
0.3
0.6 0.5 0.4 0.3
Relative error
0.8
0.7
0.6
0.3
0.2
1
0.1
0.5
Relative Performance Relative Fitness
0.2 0.9
0.8
0.7
0.6
0.5
0.3
0.2
0
0.1
0.1
0.4
Relative Relative Performance
0.2
0.4
0.3
0.1
Fraction
0.6
0
Fraction
Bandwidth (Postmark)
Relative error
Figure V.2: The cumulative distribution of relative error over all pairwise predictions.
254
Throughput (Postmark) 1 0.9 0.8 Fraction
0.7 0.6 0.5
1
0.9
0.8
0.7
0.3
0.2
0
0.1
0.1
0.6
0.2
0.5
0.3
0.4
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness
0.4
Relative error Throughput (Postmark)
1
1
0.9
0.9
0.8
0.8
0.7
0.7
Fraction
0.6 0.5 0.4
0.5 0.4 0.3
0.8
0.8
0.7
0.7
0.6 0.5 0.4
0.8
1
1 0.9
Fraction
1 0.9
1
Relative error Throughput (Postmark)
0.9
Relative error Throughput (Postmark)
0.9
0.5
0.4
0.3
0.2
0.1
0
1
0.1
0.7
Absolute Relative
0.2 0.9
0.8
0.7
0.6
0.4
0.3
0.2
0
0.1
0.1
0.5
Absolute (i=j) Absolute
0.2
0.6
0.3
0.6 0.5 0.4 0.3
Relative error
0.8
0.7
0.6
0.3
0.2
1
0.1
0.5
Relative Performance Relative Fitness
0.2 0.9
0.8
0.7
0.6
0.5
0.3
0.2
0
0.1
0.1
0.4
Relative Relative Performance
0.2
0.4
0.3
0.1
Fraction
0.6
0
Fraction
Throughput (Postmark)
Relative error
Figure V.3: The cumulative distribution of relative error over all pairwise predictions.
255
APPENDIX V. WORKLOADMIX MODEL TESTING ON POSTMARK SAMPLES
Latency (Postmark)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Latency (Postmark)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Latency (Postmark) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Latency (Postmark)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Latency (Postmark)
Relative error
Figure V.4: The cumulative distribution of relative error over all pairwise predictions.
256
Relative: Bandwidth (Avg )
Fraction
0.5 0.4 0.3 0.2 0.1 <-0
[[[0 [0 [0 [0 > .19 0.19 0.00 .19, .39, .58, .78, =0.9 , -0 , 0 7 0 0 0 0 .00 .19 .39) .58) .78) .97) ) )
0
[[0 [0 [0 [0 [0 > .16 0.16 .06, .28, .50, .73, .95, =1.1 ,0 7 .06 0.28 0.50 0.73 0.95 1.17 ) ) ) ) ) )
[[[0 [0 [0 [0 > .27 0.27 0.12 .04, .20, .36, .52, =0.6 , -0 , 0 8 0 0 0 0 .12 .04 .20) .36) .52) .68) ) )
<-0
[[[[0 [0 [0 > .25 0.25 0.13 0.02 .10, .22, .33, =0.4 , -0 , -0 , 0 5 0 0 0 .13 .02 .10 .22) .33) .45) ) ) )
Relative: Latency (Avg )
Fraction
Fraction
0.5
0
<-0
[[0 [0 [0 [0 [1 > .21 0.21 .06, .32, .58, .84, .10, =1.3 ,0 6 .06 0.32 0.58 0.84 1.10 1.36 ) ) ) ) ) )
[[[[[0 [0 > .35 0.35 0.25 0.15 0.06 .04, .14, =0.2 , -0 , -0 , -0 , 0 4 0 0 .25 .15 .06 .04 .14) .24) ) ) ) )
<-0
[[[0 [0 [0 [0 > .16 0.16 0.01 .14, .29, .44, .58, =0.7 , -0 , 0 3 0 0 0 0 .01 .14 .29) .44) .58) .73) ) )
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
<0
[ [ [ [ [ [ > .48 0.48 1.52 2.57 3.61 4.65 5.70 =6.7 ,1 , , , , , .52 2.57 3.61 4.65 5.70 6.74 4 ) ) ) ) ) )
Relative Performance: Latency (Avg )
0.6
0.1
<-0
Absolute: Latency (Avg )
Fraction
0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Fitness: Latency (Avg )
Fraction
Fraction
Fraction [[[[0 [0 [0 > .28 0.28 0.15 0.01 .12, .25, .39, =0.5 , -0 , -0 , 0 2 0 0 0 .15 .01 .12 .25) .39) .52) ) ) )
0.2
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Fitness: Throughput (Avg )
<-0
0.3
[[[0 [0 [0 [0 > .14 0.14 0.01 .13, .26, .40, .53, =0.6 , -0 , 0 7 0 0 0 0 .01 .13 .26) .40) .53) .67) ) )
Relative: Throughput (Avg )
Fraction
Fraction
Fraction <-0
0.4
<-0
Absolute: Throughput (Avg ) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Performance: Throughput (Avg ) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
Relative Fitness: Bandwidth (Avg ) 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative Performance: Bandwidth (Avg )
0.6 Fraction
Fraction
Absolute: Bandwidth (Avg ) 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[0 [0 [0 [0 [0 > .22 0.22 .02, .26, .50, .74, .98, =1.2 ,0 2 .02 0.26 0.50 0.74 0.98 1.22 ) ) ) ) ) )
0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[0 [0 [0 [0 > .28 0.28 0.10 .09, .27, .46, .65, =0.8 , -0 , 0 3 0 0 0 0 .10 .09 .27) .46) .65) .83) ) )
Figure V.5: The probability distribution of relative error over all pairwise predictions.
APPENDIX V. WORKLOADMIX MODEL TESTING ON POSTMARK SAMPLES
Bandwidth
Fraction
Fraction
Overall 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
, -0
80
)
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
, -0
80
)
Latency
Fraction
Fraction
Throughput 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
257
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
Figure V.6: Probability distributions of the difference in the absolute value of the relative error value−measured value (| predictedmeasured |) between the absolute and relative fitness models, over all predictions. The x value axis represent the amount by which relative error is reduced. Negative values indicate an increase in error.
258
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 11 / 3 / 34 10 / 7 / 31 8 / 4 / 36 7 / 5 / 36 9 / 6 / 33 9 / 2 / 37 14 / 8 / 26 10 / 6 / 32 12 / 6 / 30 5 / 9 / 34 7 / 4 / 37 10 / 7 / 31 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 9 / 7 / 32 14 / 6 / 28 14 / 4 / 30 9 / 2 / 37 16 / 5 / 27 11 / 5 / 32 11 / 3 / 34 20 / 10 / 18 13 / 7 / 28 11 / 4 / 33 3 / 5 / 40 17 / 5 / 26 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 28 / 3 / 17 28 / 6 / 14 15 / 5 / 28 15 / 6 / 27 17 / 10 / 21 13 / 8 / 27 21 / 1 / 26 15 / 8 / 25 13 / 7 / 28 16 / 4 / 28 14 / 7 / 27 17 / 9 / 22 nan% nan% nan%
Bandwidth AM vs. RM RM vs. RM’ 19 / 16 / 13 15 / 16 / 17 12 / 22 / 14 21 / 19 / 8 19 / 12 / 17 10 / 25 / 13 17 / 10 / 21 19 / 25 / 4 16 / 21 / 11 11 / 21 / 16 12 / 19 / 17 20 / 16 / 12 17 / 19 / 12 17 / 26 / 5 24 / 8 / 16 15 / 26 / 7 18 / 11 / 19 12 / 18 / 18 20 / 17 / 11 16 / 19 / 13 23 / 13 / 12 11 / 21 / 16 14 / 25 / 9 18 / 18 / 12 100% 0% 0% 100% 0% 0% Throughput AM vs. RM RM vs. RM’ 11 / 24 / 13 18 / 22 / 8 21 / 19 / 8 26 / 18 / 4 19 / 22 / 7 15 / 25 / 8 18 / 20 / 10 12 / 20 / 16 26 / 11 / 11 17 / 27 / 4 15 / 18 / 15 23 / 24 / 1 8 / 25 / 15 12 / 8 / 28 19 / 25 / 4 19 / 29 / 0 17 / 23 / 8 19 / 19 / 10 8 / 11 / 29 8 / 9 / 31 17 / 18 / 13 15 / 24 / 9 15 / 19 / 14 22 / 23 / 3 100% 0% 0% 100% 0% 0% Latency AM vs. RM RM vs. RM’ 12 / 36 / 0 23 / 24 / 1 8 / 32 / 8 28 / 16 / 4 20 / 26 / 2 26 / 15 / 7 20 / 20 / 8 17 / 23 / 8 21 / 25 / 2 24 / 23 / 1 17 / 14 / 17 13 / 26 / 9 13 / 27 / 8 19 / 14 / 15 15 / 20 / 13 20 / 19 / 9 16 / 19 / 13 23 / 14 / 11 16 / 13 / 19 15 / 23 / 10 14 / 18 / 16 17 / 26 / 5 15 / 21 / 12 12 / 14 / 22 100% 0% 0% 100% 0% 0%
RM’ vs. RF 22 / 21 / 5 20 / 24 / 4 30 / 13 / 5 14 / 26 / 8 25 / 20 / 3 18 / 21 / 9 15 / 29 / 4 15 / 24 / 9 16 / 24 / 8 20 / 23 / 5 27 / 15 / 6 21 / 23 / 4 100% 0% 0%
AM vs. RF 24 / 19 / 5 13 / 27 / 8 28 / 16 / 4 15 / 29 / 4 15 / 28 / 5 14 / 25 / 9 14 / 30 / 4 20 / 21 / 7 18 / 24 / 6 20 / 25 / 3 23 / 21 / 4 17 / 23 / 8 100% 0% 0%
RM’ vs. RF 19 / 18 / 11 17 / 27 / 4 18 / 22 / 8 33 / 11 / 4 23 / 19 / 6 19 / 23 / 6 29 / 15 / 4 18 / 23 / 7 17 / 27 / 4 23 / 17 / 8 28 / 15 / 5 26 / 19 / 3 100% 0% 0%
AM vs. RF 16 / 27 / 5 22 / 22 / 4 15 / 30 / 3 31 / 13 / 4 21 / 23 / 4 14 / 26 / 8 23 / 20 / 5 18 / 28 / 2 21 / 23 / 4 21 / 22 / 5 21 / 22 / 5 26 / 19 / 3 100% 0% 0%
RM’ vs. RF 23 / 19 / 6 14 / 34 / 0 19 / 25 / 4 30 / 15 / 3 23 / 23 / 2 25 / 21 / 2 20 / 23 / 5 25 / 23 / 0 17 / 25 / 6 26 / 19 / 3 20 / 20 / 8 19 / 25 / 4 100% 0% 0%
AM vs. RF 7 / 37 / 4 7 / 38 / 3 15 / 30 / 3 25 / 20 / 3 19 / 26 / 3 17 / 28 / 3 17 / 28 / 3 22 / 22 / 4 16 / 28 / 4 21 / 24 / 3 14 / 31 / 3 15 / 26 / 7 100% 0% 0%
Table V.2: Models are compared across all pairwise predictions. For each prediction, a score of 1 is assigned to the model with the lowest accuracy. If the accuracy of two models is within 0.01, a score of 1 goes to the tie category (3rd number shown). Hypotheses are tested in turn. Specifically, idealized absolute models (i = j) are compared against in-practice absolute models (i 6= j), relative models are compared to absolute models (AM vs. RM), relative models that use performance are compared to those that do not (RM vs. RM’), and relative fitness models are compared to relative models that use performance (RM’ vs. RF). In addition, relative fitness models are compared directly to absolute models (AM vs. RF).
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.07 0.09 0.08 0.07 0.08
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.10 0.10 0.13 0.10
Relative Array A Array B Array C Array D
A 0.10 0.12 0.11 0.13
Relative Performance Array A Array B Array C Array D
A 0.01 0.07 0.08 0.08
Relative Fitness Array A Array B Array C Array D
A 0.0 0.06 0.04 0.06
Bandwidth 0.07 0.08 0.09 0.07 0.06
Pairwise Array A Array 0.07 0.11 0.07 0.07 0.11 0.09 0.08 0.08 Array A Array 0.07 0.07 0.08 0.07 0.09 0.11 0.09 0.07 Array A Array 0.01 0.08 0.06 0.01 0.08 0.08 0.07 0.06 Array A Array 0.0 0.08 0.08 0.0 0.07 0.08 0.07 0.08
Bandwidth B C 0.05 0.09 0.05 0.09 0.05 0.08 0.05 0.08 Bandwidth B C 0.07 0.05 0.05 0.07 0.11 0.08 0.07 0.05 Bandwidth B C 0.06 0.05 0.01 0.06 0.07 0.01 0.06 0.05 Bandwidth B C 0.06 0.05 0.0 0.07 0.05 0.0 0.07 0.06
D 0.06 0.06 0.07 0.06
A 0.04 0.05 0.06 0.06
D 0.10 0.05 0.10 0.06
A 0.04 0.04 0.05 0.04
D 0.06 0.07 0.07 0.01
A 0.00 0.04 0.04 0.04
D 0.08 0.05 0.06 0.0
A 0.0 0.07 0.07 0.05
B
B
B
B
Throughput 0.06 0.08 0.07 0.06 0.07
Array 0.12 0.10 0.08 0.09 Array 0.09 0.12 0.08 0.08 Array 0.09 0.08 0.01 0.08 Array 0.07 0.09 0.0 0.09
Throughput B C 0.06 0.10 0.08 0.10 0.09 0.07 0.07 0.10 Throughput B C 0.06 0.10 0.08 0.15 0.12 0.07 0.06 0.09 Throughput B C 0.06 0.10 0.01 0.09 0.08 0.01 0.05 0.09 Throughput B C 0.05 0.09 0.0 0.07 0.07 0.0 0.09 0.10
Table V.3: Median relative error
C
C
C
C
Latency 0.08 0.13 0.10 0.10 0.10
Array 0.10 0.08 0.08 0.06 Array 0.09 0.07 0.07 0.06 Array 0.07 0.07 0.09 0.01 Array 0.08 0.07 0.07 0.0
D 0.13 0.08 0.07 0.06
A 0.07 0.10 0.15 0.12
D 0.10 0.06 0.07 0.06
A 0.07 0.09 0.11 0.12
D 0.06 0.06 0.07 0.01
A 0.01 0.07 0.13 0.09
D 0.05 0.07 0.07 0.0
A 0.0 0.13 0.09 0.13
D
D
D
D
Latency B C 0.44 0.52 0.09 0.12 0.11 0.08 0.09 0.10 Latency B C 0.11 0.12 0.09 0.11 0.09 0.08 0.09 0.09 Latency B C 0.10 0.15 0.01 0.10 0.09 0.01 0.06 0.10 Latency B C 0.13 0.08 0.0 0.13 0.10 0.0 0.07 0.10
D 0.13 0.12 0.10 0.08 D 0.09 0.10 0.07 0.08 D 0.09 0.07 0.10 0.01 D 0.09 0.07 0.09 0.0
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.20 0.33 0.21 0.18 0.17
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.16 0.18 0.17 0.13
Relative Array A Array B Array C Array D
A 0.16 0.29 0.20 0.25
Relative Performance Array A Array B Array C Array D
A 0.01 0.16 0.20 0.18
Relative Fitness Array A Array B Array C Array D
A 0.0 0.21 0.09 0.16
Bandwidth 0.15 0.17 0.23 0.15 0.14
Pairwise Array A Array 0.12 0.63 0.16 0.25 0.16 0.23 0.14 0.22 Array A Array 0.12 0.31 0.19 0.25 0.15 0.24 0.21 0.29 Array A Array 0.01 0.15 0.14 0.03 0.18 0.21 0.15 0.19 Array A Array 0.0 0.19 0.19 0.0 0.13 0.23 0.16 0.15
Bandwidth B C 0.21 0.23 0.10 0.21 0.10 0.18 0.10 0.19 Bandwidth B C 0.35 0.16 0.10 0.13 0.26 0.18 0.45 0.14 Bandwidth B C 0.10 0.10 0.01 0.25 0.17 0.01 0.09 0.27 Bandwidth B C 0.17 0.09 0.0 0.16 0.13 0.0 0.13 0.11
D 0.14 0.15 0.18 0.14
A 0.07 0.08 0.09 0.07
D 0.21 0.13 0.25 0.14
A 0.07 0.08 0.09 0.11
D 0.11 0.10 0.09 0.01
A 0.01 0.14 0.16 0.13
D 0.24 0.10 0.12 0.0
A 0.0 0.17 0.15 0.14
B
B
B
B
Throughput 0.16 0.15 0.13 0.16 0.14
Array 0.53 0.24 0.21 0.23 Array 0.17 0.20 0.21 0.17 Array 0.18 0.25 0.01 0.19 Array 0.11 0.20 0.0 0.13
Throughput B C 0.13 0.24 0.18 0.21 0.15 0.22 0.17 0.26 Throughput B C 0.09 0.16 0.18 0.19 0.14 0.22 0.10 0.18 Throughput B C 0.12 0.15 0.07 0.32 0.13 0.01 0.14 0.18 Throughput B C 0.18 0.11 0.0 0.21 0.12 0.0 0.13 0.15
Table V.4: Mean relative error
C
C
C
C
Latency 0.29 0.66 0.26 0.24 0.22
Array 0.94 0.22 0.24 0.20 Array 0.18 0.16 0.22 0.20 Array 0.20 0.12 0.23 0.02 Array 0.25 0.11 0.14 0.0
D 0.18 0.15 0.12 0.17
A 0.12 0.21 0.21 0.21
D 0.16 0.11 0.13 0.17
A 0.12 0.19 0.17 0.25
D 0.14 0.13 0.13 0.02
A 0.02 0.12 0.17 0.14
D 0.18 0.10 0.09 0.0
A 0.0 0.19 0.15 0.17
D
D
D
D
Latency B C 1.55 1.12 0.48 0.31 0.44 0.24 0.39 0.25 Latency B C 0.47 0.19 0.48 0.27 0.33 0.24 0.32 0.19 Latency B C 0.23 0.29 0.01 0.17 0.33 0.03 0.33 0.13 Latency B C 0.24 0.14 0.0 0.22 0.43 0.0 0.20 0.15
D 2.51 0.35 0.41 0.31 D 0.18 0.24 0.28 0.31 D 0.36 0.15 0.47 0.02 D 0.33 0.14 0.23 0.0
Appendix W
WorkloadMix model testing on Cello samples Application (srt) Total used
Samples 78
Iters 3
First sample 39 39
Last sample 77
Table W.1: Multiple testing samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
261
262
Overall (Cello)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
0.9
1 1
0.8
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Relative Performance Relative Fitness 0
1
0.9
0.8
0.7
0.6
0.5
Relative error
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
1
0.9
0.8
0.7
0.6
0.5
Relative error Overall (Cello)
0.4
0.3
Absolute Relative
Relative error
Relative Relative Performance 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Overall (Cello)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Overall (Cello)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Overall (Cello)
Relative error
Figure W.1: The cumulative distribution of relative error over all pairwise predictions.
263
APPENDIX W. WORKLOADMIX MODEL TESTING ON CELLO SAMPLES
Bandwidth (Cello)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Bandwidth (Cello)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Bandwidth (Cello) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Bandwidth (Cello)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Bandwidth (Cello)
Relative error
Figure W.2: The cumulative distribution of relative error over all pairwise predictions.
264
Throughput (Cello)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Throughput (Cello)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Throughput (Cello) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Throughput (Cello)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Throughput (Cello)
Relative error
Figure W.3: The cumulative distribution of relative error over all pairwise predictions.
265
APPENDIX W. WORKLOADMIX MODEL TESTING ON CELLO SAMPLES
Latency (Cello)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
0.9
1 1
0.8
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Relative Performance Relative Fitness 0
1
0.9
0.8
0.7
0.6
0.5
Relative error
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
1
0.9
0.8
0.7
0.6
0.5
Relative error Latency (Cello)
0.4
0.3
Absolute Relative
Relative error
Relative Relative Performance 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Latency (Cello)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Latency (Cello)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Latency (Cello)
Relative error
Figure W.4: The cumulative distribution of relative error over all pairwise predictions.
266
Relative: Bandwidth (Avg )
Fraction
0.2 0.15 0.1 0.05 0
<-0
[[[[0 [0 [0 > .53 0.53 0.33 0.13 .07, .27, .47, =0.6 , -0 , -0 , 0 7 0 0 0 .33 .13 .07 .27) .47) .67) ) ) )
<-0
[[[0 [0 [0 [0 > .25 0.25 0.08 .10, .28, .46, .64, =0.8 , -0 , 0 2 0 0 0 0 .08 .10 .28) .46) .64) .82) ) )
<-0
[[[0 [0 [0 [0 > .26 0.26 0.13 .00, .14, .27, .40, =0.5 , -0 , 0 4 0 0 0 0 .13 .00 .14) .27) .40) .54) ) )
<-0
[[[[0 [0 [0 > .35 0.35 0.21 0.06 .08, .23, .38, =0.5 , -0 , -0 , 0 2 0 0 0 .21 .06 .08 .23) .38) .52) ) ) )
[[[[[0 [0 > .55 0.55 0.39 0.23 0.08 .08, .23, =0.3 , -0 , -0 , -0 , 0 9 0 0 .39 .23 .08 .08 .23) .39) ) ) ) )
Fraction
<-0
[[[[0 [0 [0 > .46 0.46 0.30 0.14 .02, .19, .35, =0.5 , -0 , -0 , 0 1 0 0 0 .30 .14 .02 .19) .35) .51) ) ) )
Fraction <-0
[[[[0 [0 [0 > .24 0.24 0.13 0.03 .07, .18, .28, =0.3 , -0 , -0 , 0 8 0 0 0 .13 .03 .07 .18) .28) .38) ) ) )
0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[0 [0 [0 [1 [1 > .34 0.34 .06, .47, .87, .28, .68, =2.0 ,0 8 .06 0.47 0.87 1.28 1.68 2.08 ) ) ) ) ) )
Relative Performance: Latency (Avg )
Fraction [[[0 [0 [0 [0 > .49 0.49 0.22 .05, .33, .60, .87, =1.1 , -0 , 0 5 0 0 0 1 .22 .05 .33) .60) .87) .15) ) )
0.3 0.25 0.2 0.15 0.1 0.05 0
Absolute: Latency (Avg )
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Fitness: Latency (Avg )
Fraction
Fraction
Fraction <-0
<-0
[[[0 [0 [0 [0 > .27 0.27 0.10 .06, .22, .39, .55, =0.7 , -0 , 0 1 0 0 0 0 .10 .06 .22) .39) .55) .71) ) )
Relative Fitness: Throughput (Avg ) 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative: Latency (Avg ) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
Relative: Throughput (Avg )
Fraction
0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Performance: Throughput (Avg ) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.3 0.25 0.2 0.15 0.1 0.05 0
Absolute: Throughput (Avg )
Fraction
Fraction
Relative Fitness: Bandwidth (Avg ) 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Performance: Bandwidth (Avg )
0.25 Fraction
Fraction
Absolute: Bandwidth (Avg ) 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[0 [0 [0 [0 > .35 0.35 0.11 .14, .39, .64, .88, =1.1 , -0 , 0 3 0 0 0 1 .11 .14 .39) .64) .88) .13) ) )
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[0 [0 [0 [1 > .43 0.43 0.14 .15, .44, .73, .02, =1.3 , -0 , 0 1 0 0 1 1 .14 .15 .44) .73) .02) .31) ) )
Figure W.5: The probability distribution of relative error over all pairwise predictions.
APPENDIX W. WORKLOADMIX MODEL TESTING ON CELLO SAMPLES
Bandwidth
Fraction
Fraction
Overall 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
, -0
80
)
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
<-1 [-1 .00 .00 ,
[-0
.60
-0.
[-0 [0. [0. >= 20 60 .20 1.0 ,0 ,0 ,0 0 .40 .80 .00 .40 ) ) ) )
, -0
80
)
Latency
Fraction
Fraction
Throughput 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
267
0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
Figure W.6: Probability distributions of the difference in the absolute value of the relative error value−measured value (| predictedmeasured |) between the absolute and relative fitness models, over all predictions. The x value axis represent the amount by which relative error is reduced. Negative values indicate an increase in error.
268
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 7 / 9 / 21 2 / 5 / 30 11 / 9 / 17 6 / 2 / 29 4 / 3 / 30 19 / 8 / 10 2 / 1 / 34 5 / 7 / 25 13 / 9 / 15 3 / 2 / 32 8 / 10 / 19 3 / 4 / 30 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 18 / 7 / 12 6 / 8 / 23 3 / 9 / 25 12 / 6 / 19 10 / 2 / 25 7 / 9 / 21 10 / 7 / 20 7 / 6 / 24 9 / 9 / 19 8 / 3 / 26 17 / 7 / 13 8 / 9 / 20 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 19 / 7 / 11 14 / 8 / 15 3 / 4 / 30 10 / 4 / 23 15 / 5 / 17 20 / 4 / 13 10 / 3 / 24 11 / 4 / 22 19 / 3 / 15 7 / 2 / 28 19 / 7 / 11 19 / 7 / 11 nan% nan% nan%
Bandwidth AM vs. RM RM vs. RM’ 13 / 17 / 7 15 / 17 / 5 21 / 15 / 1 19 / 17 / 1 18 / 16 / 3 15 / 20 / 2 12 / 17 / 8 20 / 11 / 6 15 / 13 / 9 8 / 22 / 7 18 / 19 / 0 16 / 21 / 0 15 / 12 / 10 21 / 16 / 0 22 / 15 / 0 13 / 21 / 3 12 / 20 / 5 12 / 23 / 2 16 / 17 / 4 15 / 20 / 2 23 / 13 / 1 13 / 21 / 3 19 / 18 / 0 8 / 22 / 7 100% 0% 0% 100% 0% 0% Throughput AM vs. RM RM vs. RM’ 20 / 13 / 4 10 / 26 / 1 8 / 11 / 18 37 / 0 / 0 23 / 13 / 1 13 / 19 / 5 14 / 18 / 5 22 / 13 / 2 34 / 3 / 0 1 / 36 / 0 19 / 18 / 0 21 / 16 / 0 22 / 15 / 0 10 / 20 / 7 21 / 14 / 2 11 / 16 / 10 22 / 15 / 0 15 / 21 / 1 14 / 18 / 5 14 / 19 / 4 17 / 13 / 7 10 / 15 / 12 16 / 14 / 7 19 / 15 / 3 100% 0% 0% 100% 0% 0% Latency AM vs. RM RM vs. RM’ 8 / 23 / 6 15 / 22 / 0 13 / 18 / 6 16 / 19 / 2 14 / 18 / 5 18 / 12 / 7 18 / 11 / 8 12 / 19 / 6 21 / 15 / 1 17 / 20 / 0 12 / 18 / 7 11 / 19 / 7 7 / 7 / 23 11 / 9 / 17 12 / 21 / 4 16 / 21 / 0 9 / 22 / 6 27 / 6 / 4 15 / 17 / 5 19 / 11 / 7 9 / 26 / 2 19 / 14 / 4 10 / 20 / 7 22 / 15 / 0 100% 0% 0% 100% 0% 0%
RM’ vs. RF 15 / 22 / 0 9 / 27 / 1 18 / 17 / 2 17 / 17 / 3 18 / 14 / 5 17 / 17 / 3 10 / 25 / 2 12 / 23 / 2 21 / 15 / 1 13 / 21 / 3 17 / 19 / 1 20 / 16 / 1 100% 0% 0%
AM vs. RF 15 / 19 / 3 12 / 23 / 2 12 / 25 / 0 16 / 18 / 3 12 / 22 / 3 12 / 23 / 2 14 / 20 / 3 10 / 27 / 0 13 / 21 / 3 16 / 20 / 1 14 / 22 / 1 16 / 18 / 3 100% 0% 0%
RM’ vs. RF 19 / 15 / 3 0 / 37 / 0 13 / 20 / 4 17 / 17 / 3 18 / 17 / 2 15 / 21 / 1 13 / 20 / 4 15 / 18 / 4 8 / 26 / 3 13 / 20 / 4 19 / 15 / 3 18 / 15 / 4 100% 0% 0%
AM vs. RF 11 / 22 / 4 16 / 21 / 0 15 / 20 / 2 18 / 16 / 3 15 / 20 / 2 16 / 17 / 4 12 / 24 / 1 15 / 19 / 3 13 / 19 / 5 12 / 22 / 3 16 / 20 / 1 19 / 15 / 3 100% 0% 0%
RM’ vs. RF 21 / 15 / 1 14 / 23 / 0 18 / 17 / 2 24 / 13 / 0 20 / 17 / 0 17 / 17 / 3 18 / 19 / 0 20 / 12 / 5 15 / 21 / 1 14 / 19 / 4 21 / 14 / 2 19 / 18 / 0 100% 0% 0%
AM vs. RF 11 / 25 / 1 16 / 21 / 0 10 / 24 / 3 24 / 12 / 1 21 / 14 / 2 9 / 28 / 0 20 / 16 / 1 15 / 22 / 0 11 / 25 / 1 14 / 21 / 2 14 / 23 / 0 12 / 24 / 1 100% 0% 0%
Table W.2: Models are compared across all pairwise predictions. For each prediction, a score of 1 is assigned to the model with the lowest accuracy. If the accuracy of two models is within 0.01, a score of 1 goes to the tie category (3rd number shown). Hypotheses are tested in turn. Specifically, idealized absolute models (i = j) are compared against in-practice absolute models (i 6= j), relative models are compared to absolute models (AM vs. RM), relative models that use performance are compared to those that do not (RM vs. RM’), and relative fitness models are compared to relative models that use performance (RM’ vs. RF). In addition, relative fitness models are compared directly to absolute models (AM vs. RF).
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.16 0.18 0.19 0.17 0.15
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.12 0.17 0.14 0.14
Relative Array A Array B Array C Array D
A 0.12 0.11 0.12 0.14
Relative Performance Array A Array B Array C Array D
A 0.02 0.17 0.16 0.10
Relative Fitness Array A Array B Array C Array D
A 0.0 0.15 0.10 0.07
Bandwidth 0.17 0.18 0.20 0.17 0.14
Pairwise Array A Array 0.12 0.24 0.15 0.16 0.13 0.19 0.14 0.24 Array A Array 0.12 0.24 0.15 0.16 0.14 0.23 0.14 0.22 Array A Array 0.01 0.19 0.13 0.02 0.13 0.16 0.12 0.19 Array A Array 0.0 0.15 0.15 0.0 0.09 0.14 0.10 0.20
Bandwidth B C 0.22 0.15 0.20 0.17 0.19 0.17 0.24 0.16 Bandwidth B C 0.30 0.18 0.20 0.20 0.26 0.17 0.30 0.18 Bandwidth B C 0.25 0.25 0.03 0.13 0.15 0.02 0.22 0.12 Bandwidth B C 0.16 0.12 0.0 0.13 0.10 0.0 0.19 0.14
D 0.18 0.24 0.18 0.16
A 0.10 0.12 0.10 0.12
D 0.15 0.24 0.16 0.16
A 0.10 0.13 0.16 0.15
D 0.13 0.19 0.12 0.01
A 0.01 0.11 0.11 0.09
D 0.14 0.15 0.14 0.0
A 0.0 0.11 0.06 0.08
B
B
B
B
Throughput 0.12 0.13 0.16 0.13 0.09
Array 0.16 0.20 0.16 0.17 Array 0.16 0.32 0.16 0.15 Array 0.36 0.15 0.01 0.15 Array 0.12 0.18 0.0 0.17
Throughput B C 0.13 0.10 0.07 0.16 0.09 0.10 0.13 0.10 Throughput B C 0.15 0.11 0.07 0.45 0.13 0.10 0.12 0.12 Throughput B C 0.08 0.67 0.01 0.11 0.08 0.02 0.09 0.10 Throughput B C 0.06 0.07 0.0 0.10 0.09 0.0 0.10 0.15
Table W.3: Median relative error
C
C
C
C
Latency 0.17 0.24 0.20 0.20 0.22
Array 0.18 0.24 0.21 0.18 Array 0.17 0.22 0.19 0.18 Array 0.13 0.18 0.20 0.02 Array 0.14 0.16 0.15 0.0
D 0.18 0.19 0.14 0.19
A 0.13 0.14 0.14 0.14
D 0.20 0.15 0.21 0.19
A 0.13 0.17 0.14 0.12
D 0.11 0.18 0.17 0.02
A 0.01 0.13 0.13 0.15
D 0.12 0.14 0.09 0.0
A 0.0 0.21 0.17 0.15
D
D
D
D
Latency B C 0.59 0.21 0.28 0.23 0.35 0.16 0.57 0.23 Latency B C 0.26 0.17 0.28 0.29 0.25 0.16 0.24 0.17 Latency B C 0.32 0.21 0.02 0.23 0.19 0.01 0.27 0.22 Latency B C 0.32 0.18 0.0 0.25 0.26 0.0 0.30 0.26
D 0.18 0.37 0.31 0.17 D 0.13 0.20 0.20 0.17 D 0.13 0.15 0.24 0.02 D 0.14 0.17 0.20 0.0
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.22 0.33 0.30 0.26 0.25
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.18 0.22 0.18 0.18
Relative Array A Array B Array C Array D
A 0.18 0.18 0.21 0.21
Relative Performance Array A Array B Array C Array D
A 0.03 0.25 0.21 0.16
Relative Fitness Array A Array B Array C Array D
A 0.0 0.16 0.16 0.13
Bandwidth 0.22 0.25 0.29 0.23 0.20
Pairwise Array A Array 0.16 0.46 0.21 0.28 0.20 0.38 0.24 0.60 Array A Array 0.16 0.45 0.21 0.28 0.21 0.35 0.19 0.33 Array A Array 0.02 0.26 0.22 0.02 0.18 0.24 0.18 0.29 Array A Array 0.0 0.25 0.20 0.0 0.16 0.24 0.15 0.30
Bandwidth B C 0.32 0.18 0.30 0.22 0.29 0.21 0.33 0.19 Bandwidth B C 0.53 0.20 0.30 0.26 0.44 0.21 0.49 0.21 Bandwidth B C 0.32 0.27 0.04 0.17 0.30 0.02 0.35 0.16 Bandwidth B C 0.23 0.14 0.0 0.31 0.21 0.0 0.23 0.18
D 0.23 0.33 0.26 0.21
A 0.14 0.22 0.21 0.17
D 0.22 0.26 0.22 0.21
A 0.14 0.17 0.18 0.16
D 0.17 0.23 0.16 0.03
A 0.03 0.16 0.13 0.12
D 0.27 0.22 0.18 0.0
A 0.0 0.13 0.09 0.11
B
B
B
B
Throughput 0.17 0.19 0.24 0.21 0.14
Array 0.29 0.33 0.20 0.37 Array 0.29 0.47 0.20 0.34 Array 0.45 0.31 0.02 0.31 Array 0.25 0.42 0.0 0.32
Throughput B C 0.19 0.15 0.12 0.21 0.14 0.16 0.18 0.16 Throughput B C 0.33 0.16 0.12 0.46 0.18 0.16 0.23 0.17 Throughput B C 0.11 0.66 0.01 0.15 0.14 0.03 0.16 0.15 Throughput B C 0.11 0.10 0.0 0.19 0.12 0.0 0.18 0.15
Table W.4: Mean relative error
C
C
C
C
Latency 0.25 0.54 0.39 0.34 0.42
Array 0.22 0.32 0.28 0.23 Array 0.23 0.28 0.29 0.23 Array 0.19 0.23 0.26 0.03 Array 0.19 0.34 0.21 0.0
D 0.20 0.23 0.21 0.28
A 0.15 0.20 0.20 0.36
D 0.30 0.23 0.29 0.28
A 0.15 0.28 0.24 0.21
D 0.24 0.23 0.25 0.03
A 0.01 0.25 0.20 0.26
D 0.16 0.21 0.16 0.0
A 0.0 0.32 0.23 0.21
D
D
D
D
Latency B C 0.87 0.53 0.42 0.56 0.71 0.22 1.28 0.76 Latency B C 0.50 0.52 0.42 0.70 0.42 0.22 0.27 0.63 Latency B C 0.35 0.42 0.02 0.61 0.28 0.01 0.36 0.62 Latency B C 0.42 0.52 0.0 0.75 0.38 0.0 0.48 0.64
D 0.22 0.40 0.39 0.19 D 0.18 0.36 0.35 0.19 D 0.17 0.22 0.36 0.02 D 0.16 0.58 0.29 0.0
Appendix X
WorkloadMix model testing on TPC-C samples Application TPC-C (tpcc) Total used
Samples 50
Iters 3
First sample 25 25
Last sample 49
Table X.1: Multiple testing samples are collected. Each sample is run for multiple iterations in order to calculate the variance across runs.
271
272
Overall (TPC-C)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
0.9
1 1
0.8
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Relative Performance Relative Fitness 0
1
0.9
0.8
0.7
0.6
0.5
Relative error
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
1
0.9
0.8
0.7
0.6
0.5
Relative error Overall (TPC-C)
0.4
0.3
Absolute Relative
Relative error
Relative Relative Performance 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Overall (TPC-C)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Overall (TPC-C)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Overall (TPC-C)
Relative error
Figure X.1: The cumulative distribution of relative error over all pairwise predictions.
273
APPENDIX X. WORKLOADMIX MODEL TESTING ON TPC-C SAMPLES
Bandwidth (TPC-C)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error Bandwidth (TPC-C)
Relative error
0.9
1 1
0.8
0.9
0.7
0.6
0.5
0.4
0.8
0.7
Relative Performance Relative Fitness 0.6
1
0.9
0.8
0.7
0.6
0.5
0.3
0.2
0.1
0.4
Relative Relative Performance
0.2
0.5
0.3
0.4
0.4
0.3
0.5
0.2
0.6
0.1
Fraction
0.7
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
0.8
0
0.3
Relative error Bandwidth (TPC-C)
1
0.1
0.2
Relative error Bandwidth (TPC-C) 0.9
Fraction
Absolute Relative 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
Absolute (i=j) Absolute
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.1
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.1
Fraction
Bandwidth (TPC-C)
Relative error
Figure X.2: The cumulative distribution of relative error over all pairwise predictions.
274
Throughput (TPC-C)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
Relative error Throughput (TPC-C)
0.9
Relative error
0.9
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Throughput (TPC-C) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Throughput (TPC-C)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Throughput (TPC-C)
Relative error
Figure X.3: The cumulative distribution of relative error over all pairwise predictions.
275
APPENDIX X. WORKLOADMIX MODEL TESTING ON TPC-C SAMPLES
Latency (TPC-C)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
Absolute (i=j) Absolute Relative Relative Performance Relative Fitness 0
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Relative error
1 1
0.8 0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
1
0.9
0.8
Relative Performance Relative Fitness 0
Fraction Relative error
0.7
0.6
0.5
0.4
0.3
Relative Relative Performance
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.9
Relative error Latency (TPC-C)
0.9
Relative error
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Absolute Relative
Latency (TPC-C) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.2
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0
0.1
Absolute (i=j) Absolute
0
Fraction
Latency (TPC-C)
Fraction
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0.1
Fraction
Latency (TPC-C)
Relative error
Figure X.4: The cumulative distribution of relative error over all pairwise predictions.
276
<-0
[[[[0 [0 [0 > .24 0.24 0.16 0.07 .01, .09, .18, =0.2 , -0 , -0 , 0 6 0 0 0 .16 .07 .01 .09) .18) .26) ) ) )
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[[0 [0 [0 [0 > .13 0.13 0.00 .13, .25, .38, .51, =0.6 , -0 , 0 3 0 0 0 0 .00 .13 .25) .38) .51) .63) ) )
Fraction
0.4 0.3 0.2 0.1
<-0
[[[[0 [0 [0 > .17 0.17 0.09 0.01 .08, .16, .24, =0.3 , -0 , -0 , 0 2 0 0 0 .09 .01 .08 .16) .24) .32) ) ) )
0
[[[[[0 [0 > .35 0.35 0.25 0.15 0.05 .06, .16, =0.2 , -0 , -0 , -0 , 0 6 0 0 .25 .15 .05 .06 .16) .26) ) ) ) )
[ [ [ [ [ [ > .04 0.04 0.37 0.70 1.03 1.36 1.70 =2.0 ,0 , , , , , .37 0.70 1.03 1.36 1.70 2.03 3 ) ) ) ) ) )
Fraction
<-0
[[[0 [0 [0 [0 > .15 0.15 0.04 .06, .17, .28, .39, =0.4 , -0 , 0 9 0 0 0 0 .04 .06 .17) .28) .39) .49) ) )
Absolute: Latency (Avg ) 0.6 Fraction
0.5 0.4 0.3 0.2 0.1 <-0
[[[[0 [0 [0 > .17 0.17 0.10 0.04 .03, .09, .16, =0.2 , -0 , -0 , 0 2 0 0 0 .10 .04 .03 .09) .16) .22) ) ) )
0
<-0
[[0 [0 [0 [1 [1 > .32 0.32 .07, .46, .85, .24, .64, =2.0 ,0 3 .07 0.46 0.85 1.24 1.64 2.03 ) ) ) ) ) )
Relative Performance: Latency (Avg )
Fraction [[[0 [0 [0 [0 > .27 0.27 0.08 .12, .31, .51, .70, =0.9 , -0 , 0 0 0 0 0 0 .08 .12 .31) .51) .70) .90) ) )
0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<0
0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Fitness: Latency (Avg )
Fraction
Fraction
Fraction <-0
<-0
[[[[0 [0 [0 > .19 0.19 0.12 0.05 .02, .09, .16, =0.2 , -0 , -0 , 0 3 0 0 0 .12 .05 .02 .09) .16) .23) ) ) )
Relative Fitness: Throughput (Avg ) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06
Relative: Latency (Avg ) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
Relative: Throughput (Avg )
0.5
Relative Performance: Throughput (Avg ) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.24 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04
Absolute: Throughput (Avg ) 0.6
Fraction
Fraction
Relative Fitness: Bandwidth (Avg ) 0.3 0.25 0.2 0.15 0.1 0.05 0
Relative Performance: Bandwidth (Avg )
Fraction
Relative: Bandwidth (Avg )
Fraction
Fraction
Absolute: Bandwidth (Avg ) 0.22 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02
<-0
[[[[0 [0 [0 > .40 0.40 0.24 0.08 .08, .24, .40, =0.5 , -0 , -0 , 0 6 0 0 0 .24 .08 .08 .24) .40) .56) ) ) )
0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-0
[[0 [0 [1 [1 [2 > .01 0.01 .47, .96, .44, .93, .42, =2.9 ,0 0 .47 0.96 1.44 1.93 2.42 2.90 ) ) ) ) ) )
Figure X.5: The probability distribution of relative error over all pairwise predictions.
APPENDIX X. WORKLOADMIX MODEL TESTING ON TPC-C SAMPLES
Bandwidth 0.5 0.4 Fraction
Fraction
Overall 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
0.3 0.2 0.1
<-1 [-1 .00 .00 ,
[-0 -0.
.60
[-0
.20
80
, -0
)
[0.
20
.40
,0
)
.00
)
,0
[0.
60
.40
)
,0
0
>=
1.0
0
.80
)
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
Latency
Fraction
Fraction
Throughput 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
277
0.35 0.3 0.25 0.2 0.15 0.1 0.05 0
<-1 [-1 [-0 [-0 [0. [0. >= 20 60 .60 .20 1.0 .00 .00 ,0 ,0 , -0 , -0 ,0 0 .40 .80 .00 .80 .40 ) ) ) ) )
Figure X.6: Probability distributions of the difference in the absolute value of the relative error value−measured value (| predictedmeasured |) between the absolute and relative fitness models, over all predictions. The x value axis represent the amount by which relative error is reduced. Negative values indicate an increase in error.
278
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 1 / 3 / 19 5 / 2 / 16 15 / 1 / 7 2 / 1 / 20 11 / 4 / 8 17 / 1 / 5 0 / 2 / 21 1 / 3 / 19 4 / 0 / 19 2 / 1 / 20 3 / 1 / 19 1 / 0 / 22 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 4 / 6 / 13 5 / 1 / 17 11 / 5 / 7 12 / 3 / 8 11 / 1 / 11 10 / 4 / 9 14 / 5 / 4 7/8/8 6 / 2 / 15 13 / 5 / 5 12 / 8 / 3 3 / 3 / 17 nan% nan% nan%
Pair A→B A→C A→D B→A B→C B→D C→A C→B C→D D→A D→B D→C Totals
AM(i=j) vs. AM 17 / 0 / 6 1 / 0 / 22 0 / 0 / 23 3 / 0 / 20 6 / 0 / 17 2 / 1 / 20 1 / 0 / 22 21 / 0 / 2 1 / 0 / 22 1 / 0 / 22 21 / 1 / 1 4 / 1 / 18 nan% nan% nan%
Bandwidth AM vs. RM RM vs. RM’ 14 / 8 / 1 0 / 0 / 23 12 / 10 / 1 9 / 13 / 1 4 / 17 / 2 0 / 0 / 23 3 / 14 / 6 12 / 5 / 6 6 / 14 / 3 12 / 11 / 0 5 / 18 / 0 0 / 1 / 22 6 / 15 / 2 14 / 9 / 0 3 / 5 / 15 0 / 1 / 22 3 / 6 / 14 9 / 14 / 0 2 / 9 / 12 0 / 0 / 23 2 / 5 / 16 14 / 8 / 1 13 / 7 / 3 9/9/5 100% 0% 0% 100% 0% 0% Throughput AM vs. RM RM vs. RM’ 2 / 7 / 14 10 / 11 / 2 10 / 10 / 3 21 / 2 / 0 5 / 14 / 4 0 / 3 / 20 5 / 14 / 4 8 / 15 / 0 8 / 14 / 1 14 / 8 / 1 1 / 20 / 2 4 / 4 / 15 7 / 12 / 4 5 / 18 / 0 18 / 5 / 0 13 / 10 / 0 4 / 16 / 3 6 / 11 / 6 7 / 16 / 0 11 / 11 / 1 14 / 7 / 2 14 / 5 / 4 12 / 10 / 1 10 / 13 / 0 100% 0% 0% 100% 0% 0% Latency AM vs. RM RM vs. RM’ 15 / 8 / 0 9 / 13 / 1 1 / 0 / 22 1 / 1 / 21 13 / 10 / 0 12 / 10 / 1 0 / 3 / 20 6 / 16 / 1 15 / 8 / 0 16 / 7 / 0 12 / 10 / 1 13 / 10 / 0 0 / 1 / 22 9 / 13 / 1 5 / 13 / 5 3 / 4 / 16 12 / 11 / 0 11 / 12 / 0 0 / 0 / 23 12 / 9 / 2 7 / 15 / 1 13 / 10 / 0 2 / 4 / 17 15 / 8 / 0 100% 0% 0% 100% 0% 0%
RM’ vs. RF 6 / 13 / 4 13 / 9 / 1 11 / 11 / 1 7 / 15 / 1 11 / 9 / 3 12 / 9 / 2 10 / 11 / 2 17 / 3 / 3 14 / 8 / 1 11 / 12 / 0 10 / 11 / 2 9 / 12 / 2 100% 0% 0%
AM vs. RF 9 / 10 / 4 12 / 9 / 2 5 / 17 / 1 5 / 16 / 2 5 / 18 / 0 6 / 17 / 0 11 / 11 / 1 13 / 7 / 3 9 / 11 / 3 10 / 12 / 1 10 / 10 / 3 11 / 9 / 3 100% 0% 0%
RM’ vs. RF 13 / 8 / 2 1 / 22 / 0 16 / 7 / 0 7 / 13 / 3 6 / 16 / 1 14 / 8 / 1 10 / 9 / 4 12 / 9 / 2 12 / 9 / 2 11 / 12 / 0 6 / 16 / 1 14 / 8 / 1 100% 0% 0%
AM vs. RF 10 / 10 / 3 9 / 12 / 2 5 / 16 / 2 0 / 22 / 1 3 / 18 / 2 4 / 19 / 0 4 / 18 / 1 9 / 11 / 3 7 / 14 / 2 4 / 18 / 1 10 / 8 / 5 14 / 9 / 0 100% 0% 0%
RM’ vs. RF 8 / 14 / 1 11 / 11 / 1 12 / 9 / 2 19 / 4 / 0 13 / 10 / 0 19 / 1 / 3 11 / 8 / 4 18 / 5 / 0 11 / 10 / 2 15 / 8 / 0 11 / 11 / 1 9 / 11 / 3 100% 0% 0%
AM vs. RF 6 / 17 / 0 12 / 11 / 0 15 / 6 / 2 14 / 8 / 1 15 / 5 / 3 16 / 4 / 3 12 / 9 / 2 7 / 16 / 0 14 / 9 / 0 13 / 9 / 1 5 / 18 / 0 10 / 13 / 0 100% 0% 0%
Table X.2: Models are compared across all pairwise predictions. For each prediction, a score of 1 is assigned to the model with the lowest accuracy. If the accuracy of two models is within 0.01, a score of 1 goes to the tie category (3rd number shown). Hypotheses are tested in turn. Specifically, idealized absolute models (i = j) are compared against in-practice absolute models (i 6= j), relative models are compared to absolute models (AM vs. RM), relative models that use performance are compared to those that do not (RM vs. RM’), and relative fitness models are compared to relative models that use performance (RM’ vs. RF). In addition, relative fitness models are compared directly to absolute models (AM vs. RF).
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.09 0.12 0.09 0.10 0.10
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.10 0.13 0.09 0.10
Relative Array A Array B Array C Array D
A 0.10 0.09 0.07 0.10
Relative Performance Array A Array B Array C Array D
A 0.03 0.08 0.14 0.10
Relative Fitness Array A Array B Array C Array D
A 0.0 0.07 0.09 0.12
Bandwidth 0.09 0.10 0.08 0.09 0.09
Pairwise Array A Array 0.11 0.07 0.17 0.06 0.13 0.08 0.13 0.10 Array A Array 0.11 0.09 0.09 0.06 0.10 0.08 0.10 0.07 Array A Array 0.01 0.09 0.08 0.01 0.10 0.06 0.08 0.11 Array A Array 0.0 0.07 0.07 0.0 0.09 0.12 0.14 0.10
Bandwidth B C 0.06 0.09 0.06 0.12 0.06 0.09 0.06 0.09 Bandwidth B C 0.10 0.12 0.06 0.09 0.05 0.09 0.06 0.09 Bandwidth B C 0.10 0.07 0.03 0.12 0.04 0.02 0.10 0.12 Bandwidth B C 0.07 0.12 0.0 0.07 0.09 0.0 0.07 0.11
D 0.21 0.22 0.10 0.06
A 0.11 0.23 0.20 0.20
D 0.07 0.07 0.07 0.06
A 0.11 0.07 0.11 0.08
D 0.07 0.07 0.07 0.02
A 0.00 0.05 0.08 0.07
D 0.08 0.06 0.10 0.0
A 0.0 0.04 0.07 0.12
B
B
B
B
Throughput 0.08 0.13 0.08 0.08 0.08
Array 0.10 0.17 0.09 0.10 Array 0.11 0.10 0.09 0.11 Array 0.20 0.16 0.02 0.12 Array 0.12 0.11 0.0 0.12
Throughput B C 0.05 0.09 0.05 0.22 0.05 0.06 0.05 0.06 Throughput B C 0.04 0.07 0.05 0.07 0.08 0.06 0.08 0.07 Throughput B C 0.04 0.40 0.01 0.07 0.06 0.01 0.09 0.06 Throughput B C 0.05 0.07 0.0 0.06 0.07 0.0 0.07 0.11
Table X.3: Median relative error
C
C
C
C
Latency 0.10 0.13 0.14 0.14 0.17
Array 0.16 0.19 0.10 0.08 Array 0.08 0.07 0.08 0.08 Array 0.07 0.08 0.08 0.01 Array 0.12 0.11 0.10 0.0
D 0.19 0.23 0.11 0.11
A 0.09 0.12 0.11 0.11
D 0.08 0.05 0.07 0.11
A 0.09 0.09 0.09 0.11
D 0.06 0.06 0.06 0.01
A 0.02 0.10 0.10 0.09
D 0.10 0.10 0.09 0.0
A 0.0 0.24 0.08 0.17
D
D
D
D
Latency B C 0.23 0.15 0.06 0.15 0.26 0.12 0.31 0.17 Latency B C 0.26 0.17 0.06 0.24 0.23 0.12 0.17 0.17 Latency B C 0.14 0.17 0.01 0.38 0.18 0.05 0.15 0.15 Latency B C 0.08 0.12 0.0 0.29 0.23 0.0 0.14 0.12
D 0.07 0.08 0.07 0.07 D 0.08 0.08 0.09 0.07 D 0.08 0.11 0.10 0.03 D 0.13 0.18 0.09 0.0
Model Absolute (i = j) Absolute Relative Relative Performance Relative Fitness
Overall 0.17 0.32 0.18 0.16 0.23
Absolute Array A Array B Array C Array D Relative Array A Array B Array C Array D Relative Performance Array A Array B Array C Array D Relative Fitness Array A Array B Array C Array D
Absolute Array A Array B Array C Array D
A 0.13 0.27 0.11 0.13
Relative Array A Array B Array C Array D
A 0.13 0.13 0.09 0.12
Relative Performance Array A Array B Array C Array D
A 0.04 0.11 0.14 0.13
Relative Fitness Array A Array B Array C Array D
A 0.0 0.08 0.12 0.12
Bandwidth 0.11 0.16 0.13 0.11 0.12
Pairwise Array A Array 0.21 0.15 0.37 0.10 0.33 0.20 0.22 0.27 Array A Array 0.21 0.20 0.23 0.10 0.15 0.17 0.15 0.28 Array A Array 0.03 0.17 0.10 0.02 0.13 0.13 0.18 0.20 Array A Array 0.0 0.11 0.18 0.0 0.14 0.22 0.16 0.17
Bandwidth B C 0.07 0.14 0.08 0.23 0.07 0.11 0.24 0.12 Bandwidth B C 0.10 0.24 0.08 0.12 0.11 0.11 0.07 0.13 Bandwidth B C 0.10 0.12 0.04 0.13 0.06 0.03 0.10 0.13 Bandwidth B C 0.08 0.21 0.0 0.11 0.13 0.0 0.10 0.12
D 0.21 0.22 0.14 0.10
A 0.34 0.49 0.68 0.31
D 0.12 0.17 0.12 0.10
A 0.34 0.42 0.22 0.11
D 0.12 0.12 0.11 0.04
A 0.00 0.07 0.10 0.14
D 0.14 0.12 0.11 0.0
A 0.0 0.06 0.15 0.12
B
B
B
B
Throughput 0.26 0.37 0.15 0.13 0.11
Array 0.24 0.76 0.14 0.42 Array 0.19 0.19 0.14 0.14 Array 0.26 0.21 0.03 0.13 Array 0.29 0.79 0.0 0.13
Throughput B C 0.08 0.42 0.13 0.66 0.07 0.15 0.08 0.15 Throughput B C 0.05 0.11 0.13 0.12 0.12 0.15 0.11 0.13 Throughput B C 0.06 0.39 0.01 0.12 0.10 0.01 0.17 0.09 Throughput B C 0.07 0.10 0.0 0.09 0.11 0.0 0.10 0.12
Table X.4: Mean relative error
C
C
C
C
Latency 0.13 0.42 0.26 0.23 0.46
Array 0.31 0.28 0.25 0.21 Array 0.14 0.15 0.15 0.21 Array 0.17 0.12 0.12 0.03 Array 0.15 0.29 0.13 0.0
D 0.61 0.47 0.42 0.42
A 0.15 0.36 0.21 0.21
D 0.19 0.12 0.12 0.42
A 0.15 0.15 0.15 0.21
D 0.11 0.12 0.11 0.01
A 0.03 0.13 0.14 0.27
D 0.16 0.14 0.11 0.0
A 0.0 0.41 0.15 0.24
D
D
D
D
Latency B C 0.29 0.17 0.10 1.40 0.46 0.16 0.48 0.98 Latency B C 0.44 0.23 0.10 0.32 0.29 0.16 0.67 0.18 Latency B C 0.36 0.27 0.01 0.38 0.23 0.05 0.32 0.18 Latency B C 0.18 0.54 0.0 2.18 0.41 0.0 0.31 0.16
D 0.10 0.13 0.19 0.10 D 0.11 0.14 0.21 0.10 D 0.27 0.13 0.12 0.04 D 0.17 0.60 0.16 0.0
