Black-box optimization benchmarking of NIPOP-aCMA-ES and NBIPOP-aCMA-ES on the BBOB-2012 noiseless testbed Ilya Loshchilov

Marc Schoenauer

Michèle Sebag

TAO, INRIA Saclay U. Paris Sud, F-91405 Orsay

TAO, INRIA Saclay U. Paris Sud, F-91405 Orsay

CNRS, LRI UMR 8623 U. Paris Sud, F-91405 Orsay

[email protected] ABSTRACT In this paper, we study the performance of NIPOP-aCMAES and NBIPOP-aCMA-ES, recently proposed alternative restart strategies for CMA-ES. Both algorithms were tested using independent restarts till a total number of function evaluation of 106 D was reached, where D is the dimension of the function search space. We compared new strategies to CMA-ES with IPOP and BIPOP restart schemes, two algorithms with one of the best overall performance observed during the BBOB-2009 and BBOB-2010. We also present the first benchmarking of BIPOP-CMA-ES with active weighted covariance matrix update (BIPOP-aCMA-ES). The comparison shows that NIPOP-aCMA-ES usually outperforms IPOP-aCMA-ES and has similar performance with BIPOP-aCMA-ES, while using only the regime of increasing the population size. The second strategy, NBIPOPaCMA-ES, outperforms BIPOP-aCMA-ES in dimension 40 on weakly structured multi-modal functions thanks to adaptive allocation of computation budgets between the regimes of restarts.

Categories and Subject Descriptors G.1.6 [Numerical Analysis]: Optimization—global optimization, unconstrained optimization; F.2.1 [Analysis of Algorithms and Problem Complexity]: Numerical Algorithms and Problems

General Terms Algorithms

Keywords Benchmarking, black-box optimization, evolution strategy, CMA-ES, self-adaptation, restart strategies

1. INTRODUCTION

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. GECCO’12, July 7–11, 2012, Philadelphia, USA. Copyright 2012 ACM 978-1-4503-0073-5/10/07 ...$10.00.

The CMA-ES algorithm is a stochastic optimizer, searching the continuous space RD by sampling λ candidate solutions from a multivariate normal distribution [10, 9]. It exploits the best µ solutions out of the λ ones to adaptively estimate the local covariance matrix of the objective function, in order to increase the probability of successful samples in the next iteration. The information about the remaining (worst λ − µ) solutions is used only implicitly during the selection process. The CMA-ES has been extended to active (µ/µI , λ)-CMAES [12] and weighted active (µ/µw , λ)-CMA-ES (aCMA-ES [11]), where the information about worst λ − µ points can be also taken into account to reduce the variance of the mutation distribution in unpromising directions. However, aCMA-ES no longer guarantees the positive-definiteness of the covariance matrix, possibly resulting in algorithmic instability. The instability issues can however be numerically controlled during the search; as a matter of fact they are never observed on the BBOB benchmark suite. Two versions of CMA-ES with restarts have been proposed to handle multi-modal functions: IPOP-CMA-ES [1] was ranked first on the continuous optimization benchmark at CEC 2005 [4, 3]; and BIPOP-CMA-ES [5] showed the best results together with IPOP-CMA-ES on the black-box optimization benchmark (BBOB) in 2009 and 2010. The restart strategies of CMA-ES can be viewed as a noisy optimization problem of proper hyper-parameters of the CMA-ES in a 2D space (population size, initial stepsize). In this paper we study the performance of two alternative restart strategies for CMA-ES, NIPOP-aCMA-ES and NBIPOP-aCMA-ES. The interested reader is referred to [13] for an in-depth presentation and discussion of these algorithms.

2. 2.1

THE ALGORITHMS The IPOP-aCMA-ES

A search for the global optima of multimodal function can be difficult if the number of local optima is high. For the specific case of the CMA-ES algorithm it has been observed that the probability and the overall number of function evaluations to reach the optima are very sensitive to the population size [9]. The default population size λdef ault , tuned for uni-modal functions, is not sufficiently large for multi-modal functions. This observation led to an idea to restart the CMA-ES, each time with larger population size [1] to perform a more global search. The restart (µ/µw , λ)-CMA-ES

with increasing population (IPOP-CMA-ES [1]) launches independent restarts and double the population size each time at least one of the stopping criterion is met. The IPOPaCMA-ES is an extension of weighted active CMA-ES in IPOP restart scheme [11], which usually performs not worse than IPOP-CMA-ES on noiseless and noisy functions.

2.2 The BIPOP-aCMA-ES In BIPOP-CMA-ES after the first single run with default population size, we restart the algorithm in one of two possible regimes and account the budget of function evaluations spent in the corresponding regime. Each time we restart the algorithm, we use the regime with smallest budget used so far. Under the first regime we double the population size λlarge = 2irestart λdef ault in each restart irestart and use some fixed 0 0 initial step-size σlarge = σdef ault . This regime corresponds to the IPOP-CMA-ES. Under the second regime we restart the CMA-ES with 0 some small population size λsmall and step-size σsmall , where λsmall is set to  U [0,1]2   λlarge , λsmall = λdef ault 21 λdef ault

(1)

Here U [0, 1] denote independent uniformly distributed numbers in [0, 1] and λsmall ∈ [λdef ault , λ/2]. The initial step0 0 −2U [0,1] size is set to σsmall = σdef . ault × 10 In each restart, BIPOP-CMA-ES selects the restart regime with less function evaluations. Clearly, the second regime consumes less function evaluations than the doubling regime; it is therefore launched more often. The BIPOP-aCMA-ES is an extension of BIPOP-CMAES to the case of weighted active covariance matrix update (weighted active (µ/µw , λ)-CMA-ES in BIPOP restart scheme), which will be for the first time benchmarked in this paper.

2.3 The NIPOP-aCMA-ES In NIPOP-aCMA-ES in addition to increasing of population size in each restart, we also decrease the initial step-size by some factor kσdec . In this study we choose kσdec = 1.6 such that σ value after 9 restarts roughly corresponds to the minimum possible initial σ = 10−2 σdef ault used for BIPOPCMA-ES.

2.4 The NBIPOP-aCMA-ES In NBIPOP-aCMA-ES as well as in BIPOP-aCMA-ES we have two restart regimes: i). Double the population size and decrease the initial stepsize by kσdec = 1.6 (NIPOP-aCMA-ES). ii). Launch CMA-ES with default population size λdef ault 0 −2U [0,1] and σ 0 = σdef . ault × 10 In contrast with BIPOP-CMA-ES, where both regimes have the same budget, the budget is adapted here according to the performance of the regime: the best solutions x∗A and x∗B found by regimes A and B are used as an estimator of their quality. We thus allocate kbudget = 2 times larger computation budget for regime A if it performs better than B (i.e., if x∗A is better than x∗B ), and vice versa.

2.5 The Benchmarked Algorithms For benchmarking we consider four CMA-ES algorithms in restart scenario: IPOP-aCMA-ES [11], BIPOP-aCMA-ES

as an extension of BIPOP-CMA-ES [6], NIPOP-aCMA-ES and NBIPOP-aCMA-ES [13]. In (µ/µw , λ)-CMA-ES part of these algorithms we use default parameters as given in [11] and [6]. The maximum budget of function evaluations is 106 D and 0 the initial step-size σdef ault = 2.0.

3.

RESULTS

Results from experiments according to [7] on the benchmark functions given in [2, 8] are presented in Figures 1, 2 and 3 and in Tables 1 and 2. The expected running time (ERT), used in the figures and table, depends on a given target function value, ft = fopt + ∆f , and is computed over all relevant trials as the number of function evaluations executed during each trial while the best function value did not reach ft , summed over all trials and divided by the number of trials that actually reached ft [7, 14]. Statistical significance is tested with the rank-sum test for a given target ∆ft (10−8 as in Figure 1) using, for each trial, either the number of needed function evaluations to reach ∆ft (inverted and multiplied by −1), or, if the target was not reached, the best ∆f -value achieved, measured only up to the smallest number of overall function evaluations for any unsuccessful trial under consideration. All benchmarked here algorithms represent (µ/µw , λ)-CMAES before the first restart occurs, therefore, the results are very similar for the uni-modal functions, where the optimum can be found without restarts. We show the results in 40-D instead of 20-D, because the difference between algorithms is more significant in higher dimensions. NIPOP-aCMA-ES. On 6 out of 12 test functions (f15 , f16 , f17 , f18 , f23 , f24 ) NIPOP-aCMA-ES obtains the best known results for BBOB-2009 and BBOB-2010 workshops. On f23 Katsuuras and f24 Lunacek bi-Rastrigin, NIPOPaCMA-ES has a speedup of a factor from 2 to 3, as expected. It performs unexpectedly well on f16 Weierstrass functions, 7 times faster than IPOP-aCMA-ES and almost 3 times faster than BIPOP-aCMA-ES. Overall, according to Fig. 3, NIPOP-aCMA-ES performs as well as BIPOPaCMA-ES, while restricted to only one regime of increasing population size. NBIPOP-aCMA-ES. Thanks to the first regime of increasing population size, NBIPOP-aCMA-ES inherits some results of NIPOP-aCMA-ES. However, on functions where the population size does not play any important role, it performs significantly better than BIPOP-aCMA-ES. This is the case for f21 Gallagher 101 peaks and f22 Gallagher 21 peaks functions, where NBIPOP-aCMA-ES has a speedup of a factor of 6. It seems that the adaptive choice between two regimes works efficiently on all functions except on f16 Weierstrass, where NBIPOP-aCMA-ES incorrectly prefers small populations. This leads to a loss of a factor of 4 in comparison to NIPOP-aCMA-ES, while a factor of 1.5 is expected in the case of correct adaptation. An interesting result is comparatively good performance of NBIPOP-aCMAES on 5-dimensional f4 Skew Rastrigin Bueche multi-modal function, where NBIPOP-aCMA-ES is the only algorithm among 4 tested here, which is able to find the global optimum in 9 runs out of 15. According to Fig. 3, NBIPOPaCMA-ES performs better than BIPOP-aCMA-ES on weakly structured multi-modal functions, showing overall best results for BBOB-2009 and BBOB-2010 workshops in dimensions 20 and 40.

4. CONCLUSION

noiseless testbed. In GECCO ’10: Proceedings of the 12th annual conference comp on Genetic and In this paper, we have compared the recently proposed evolutionary computation, pages 1673–1680, New restart strategies for aCMA-ES, NIPOP-aCMA-ES and NBIPOPYork, NY, USA, 2010. ACM. aCMA-ES with the IPOP-aCMA-ES and BIPOP-aCMA[12] G. A. Jastrebski and D. V. Arnold. Improving ES. The main message of the paper is that the decreasing evolution strategies through active covariance matrix of initial step-size makes IPOP restart scenario more roadaptation. In IEEE Congress on Evolutionary bust and sometimes even comparable to BIPOP scenario on Computation – CEC 2006, pages 2814–2821, 2006. noiseless functions. We also suppose that the adaptation [13] I. Loshchilov, M. Schoenauer, and M. Sebag. of the computation budgets of different restart regimes is Alternative Restart Strategies for CMA-ES. In Proc. a promising idea for black-box optimization and should be PPSN XII, page under review, 2012. further investigated. [14] K. Price. Differential evolution vs. the functions of the second ICEO. In Proceedings of the IEEE 5. ACKNOWLEDGMENTS International Congress on Evolutionary Computation, This work was partially funded by FUI of System@tic pages 153–157, 1997. Paris-Region ICT cluster through contract DGT 117 407 Complex Systems Design Lab (CSDL).

6. REFERENCES [1] A. Auger and N. Hansen. A restart CMA evolution strategy with increasing population size. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC 2005), pages 1769–1776. IEEE Press, 2005. [2] S. Finck, N. Hansen, R. Ros, and A. Auger. Real-parameter black-box optimization benchmarking 2009: Presentation of the noiseless functions. Technical Report 2009/20, Research Center PPE, 2009. Updated February 2010. [3] S. Garc´ıa, D. Molina, M. Lozano, and F. Herrera. A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behaviour: a case study on the CEC’2005 special session on real parameter optimization. Journal of Heuristics, 15:617–644, 2009. [4] N. Hansen. Compilation of results on the 2005 CEC benchmark function set. Online, May 2006. [5] N. Hansen. Benchmarking a BI-population CMA-ES on the BBOB-2009 function testbed. In F. Rothlauf, editor, GECCO (Companion), pages 2389–2396. ACM, 2009. [6] N. Hansen. Benchmarking a BI-population CMA-ES on the BBOB-2009 function testbed. In Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers, GECCO ’09, pages 2389–2396, New York, NY, USA, 2009. ACM. [7] N. Hansen, A. Auger, S. Finck, and R. Ros. Real-parameter black-box optimization benchmarking 2012: Experimental setup. Technical report, INRIA, 2012. [8] N. Hansen, S. Finck, R. Ros, and A. Auger. Real-parameter black-box optimization benchmarking 2009: Noiseless functions definitions. Technical Report RR-6829, INRIA, 2009. Updated February 2010. [9] N. Hansen and S. Kern. Evaluating the cma evolution strategy on multimodal test functions. In PPSN’04, pages 282–291, 2004. [10] N. Hansen and A. Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation, 9(2):159–195, 2001. [11] N. Hansen and R. Ros. Benchmarking a weighted negative covariance matrix update on the BBOB-2010

1 Sphere

3

2 Ellipsoid separable

4

7

3 Rastrigin separable

7

6 3

2

1

BIPOP-aCMA IPOP-aCMA NBIPOP-aCMA NIPOP-aCMA

2 1

0

ftarget=1e-08 5 2 3 10 20 5 Linear slope 2

40

0 ftarget=1e-08 5 2 3

10

20

6 Attractive sector

3

40

1

1 0

ftarget=1e-08 5 2 3

4

0 10

20

9 Rosenbrock rotated

40

ftarget=1e-08 5 2 3

10

20

40

10 Ellipsoid

4

6

5

5

4

4

3

3

2

2

1

1

0 ftarget=1e-08 5 2 3

0 ftarget=1e-08 5 2 3

4

2

10

20

40

7 Step-ellipsoid

10

20

40

10

20

40

0 ftarget=1e-08 5 2 3

10

20

40

6 5 4 3 2 1 0 ftarget=1e-08 2 3 5

10

20

40

10

20

40

BIPOP-aCMA IPOP-aCMA NBIPOP-aCMA NIPOP-aCMA 0 ftarget=1e-08 5 2 3 10

20

40

4

3

3

2

2

1

1

0 ftarget=1e-08 5 2 3

4

10

11 Discus

20

40

4

3

3

3

2

2

2

2

1

1

1

1

5

10

20

40

13 Sharp ridge

0 ftarget=1e-08 5 2 3

4

4

10

20

40

14 Sum of different powers

5

1

5

10

20

40

17 Schaffer F7, condition 10

1

0 ftarget=1e-08 5 2 3 5

4

4

3 2

40

0 ftarget=1e-08 5 2 3 10 20 40 19 Griewank-Rosenbrock F8F2 7

5 4

2

3

3

2

2

7

10

20

22 Gallagher 21 peaks

40

1

1

0 ftarget=1e-08 5 2 3

0 ftarget=1e-08 5 2 3

6

6

5

5

4

4

3

3

2

2

2

1

1

1

10

20

40

0 ftarget=1e-08 5 2 3

10

20

40

23 Katsuuras

7 5

4

4

3

20

40

0 ftarget=1e-08 5 2 3

24 Lunacek bi-Rastrigin

6

5

10

20 Schwefel x*sin(x)

6

4

6

0 ftarget=1e-08 5 2 3

7

3

0 ftarget=1e-08 5 2 3

20

40

12 Bent cigar

16 Weierstrass

5

0 ftarget=1e-08 5 2 3

10

20

6

1

21 Gallagher 101 peaks

10

18 Schaffer F7, condition 1000

1

7

40

2

2

0 ftarget=1e-08 5 2 3

15 Rastrigin

20

3

2

1

10

4

3

3

0 ftarget=1e-08 5 2 3

8 Rosenbrock original

0 ftarget=1e-08 5 2 3

3

0 ftarget=1e-08 5 2 3

4 Skew Rastrigin-Bueche separ

3 2

1

10

20

40

Figure 1: Expected running time (ERT in number of f -evaluations) divided by dimension for target function value 10−8 as log10 values versus dimension. Different symbols correspond to different algorithms given in the legend of f1 and f24 . Light symbols give the maximum number of function evaluations from the longest trial divided by dimension. Horizontal lines give linear scaling, slanted dotted lines give quadratic scaling. Black stars indicate statistically better result compared to all other algorithms with p < 0.01 and Bonferroni correction number of dimensions (six). Legend: ◦: BIPOP-aCMA, ▽: IPOP-aCMA, ⋆: NBIPOP-aCMA, 2: NIPOP-aCMA.

separable fcts

1.0 f1-5

moderate fcts

1.0 f6-9

best2009 2009 best

Proportion of functions

Proportion of functions

best2009 2009 best

NBIPOP-aCMA NBIPOP-aCMA

0.5

BIPOP-aCMA BIPOP-aCMA

NIPOP-aCMA NIPOP-aCMA

0.0 0

1

2 3 4 5 6 7 8 log10 of (# f-evals / dimension)

IPOP-aCMA IPOP-aCMA

0.5

NIPOP-aCMA NIPOP-aCMA

0.0 0

IPOP-aCMA IPOP-aCMA

NBIPOP-aCMA NBIPOP-aCMA

1

2 3 4 5 6 7 8 log10 of (# f-evals / dimension)

ill-conditioned fcts

multi-modal fcts

1.0 f15-19

best2009 2009 best

best2009 2009 best

Proportion of functions

Proportion of functions

1.0 f10-14

NBIPOP-aCMA NBIPOP-aCMA

0.5

NIPOP-aCMA NIPOP-aCMA

IPOP-aCMA IPOP-aCMA

0.0 0

1

2 3 4 5 6 7 8 log10 of (# f-evals / dimension)

IPOP-aCMA IPOP-aCMA

0.5

NIPOP-aCMA NIPOP-aCMA

0.0 0

BIPOP-aCMA BIPOP-aCMA

BIPOP-aCMA BIPOP-aCMA

1

weakly structured multi-modal fcts

best2009 2009 best

BIPOP-aCMA BIPOP-aCMA

1

2 3 4 5 6 7 8 log10 of (# f-evals / dimension)

NBIPOP-aCMA NBIPOP-aCMA

all functions

1.0 f1-24

NIPOP-aCMA NIPOP-aCMA

NBIPOP-aCMA NBIPOP-aCMA

NBIPOP-aCMA NBIPOP-aCMA

0.5

0.0 0

2 3 4 5 6 7 8 log10 of (# f-evals / dimension)

Proportion of functions

Proportion of functions

1.0 f20-24

BIPOP-aCMA BIPOP-aCMA

IPOP-aCMA IPOP-aCMA

best2009 2009 best

0.5

0.0 0

NIPOP-aCMA NIPOP-aCMA

BIPOP-aCMA BIPOP-aCMA

1

2 3 4 5 6 7 8 log10 of (# f-evals / dimension)

IPOP-aCMA IPOP-aCMA

Figure 2: Bootstrapped empirical cumulative distribution of the number of objective function evaluations divided by dimension (FEvals/D) for 50 targets in 10[−8..2] for all functions and subgroups in 5-D. The “best 2009” line corresponds to the best ERT observed during BBOB 2009 for each single target.

separable fcts

1.0 f1-5

moderate fcts

1.0 f6-9

NIPOP-aCMA NIPOP-aCMA

0.5

BIPOP-aCMA BIPOP-aCMA

best2009 2009 best

Proportion of functions

Proportion of functions

best2009 2009 best

IPOP-aCMA IPOP-aCMA

0.5

NBIPOP-aCMA NBIPOP-aCMA

NBIPOP-aCMA NBIPOP-aCMA

0.0 0

1

2 3 4 5 6 7 8 log10 of (# f-evals / dimension)

BIPOP-aCMA BIPOP-aCMA

0.0 0

IPOP-aCMA IPOP-aCMA

1

2 3 4 5 6 7 8 log10 of (# f-evals / dimension)

ill-conditioned fcts

multi-modal fcts

NIPOP-aCMA NIPOP-aCMA

0.5

IPOP-aCMA IPOP-aCMA

BIPOP-aCMA BIPOP-aCMA

0.0 0

1

2 3 4 5 6 7 8 log10 of (# f-evals / dimension)

1.0 f15-19

best2009 2009 best

NIPOP-aCMA NIPOP-aCMA

0.5

NBIPOP-aCMA NBIPOP-aCMA

BIPOP-aCMA BIPOP-aCMA

NBIPOP-aCMA NBIPOP-aCMA

0.0 0

1

weakly structured multi-modal fcts

0.0 0

BIPOP-aCMA BIPOP-aCMA

NIPOP-aCMA NIPOP-aCMA

1

2 3 4 5 6 7 8 log10 of (# f-evals / dimension)

best best2009 2009

all functions

1.0 f1-24

NBIPOP-aCMA NBIPOP-aCMA

best2009 2009 best

0.5

2 3 4 5 6 7 8 log10 of (# f-evals / dimension)

IPOP-aCMA IPOP-aCMA

best2009 2009 best

Proportion of functions

Proportion of functions

1.0 f20-24

IPOP-aCMA IPOP-aCMA

Proportion of functions

Proportion of functions

1.0 f10-14

NIPOP-aCMA NIPOP-aCMA

NBIPOP-aCMA NBIPOP-aCMA

0.5

0.0 0

BIPOP-aCMA BIPOP-aCMA

NIPOP-aCMA NIPOP-aCMA

1

2 3 4 5 6 7 8 log10 of (# f-evals / dimension)

IPOP-aCMA IPOP-aCMA

Figure 3: Bootstrapped empirical cumulative distribution of the number of objective function evaluations divided by dimension (FEvals/D) for 50 targets in 10[−8..2] for all functions and subgroups in 40-D. The “best 2009” line corresponds to the best ERT observed during BBOB 2009 for each single target.

∆fopt 1e1 f1 11 BIPOP-a 3.1(2) IPOP-aC 3.2(3) NBIPOP-2.4(2) NIPOP-a 2.7(2)

1e0 12 8.0(3) 8.9(3) 8.3(3) 7.3(3)

∆fopt 1e1 f2 83 BIPOP-a 10(4) IPOP-aC 10(3) NBIPOP- 11(3) NIPOP-a 10(3)

1e0 87 12(3) 12(2) 12(3) 12(2)

∆fopt 1e1 1e0 f3 716 1622 12(11) BIPOP-a 1.6(2) IPOP-aC 1.1(1) 20(11) NBIPOP-0.87(0.3)13(18) NIPOP-a 1.4(1) 29(12) ∆fopt 1e1 f4 809 BIPOP-a 1.6(1) IPOP-aC 1.8(1) NBIPOP- 1.7(1) NIPOP-a 2.8(3)

1e-1 12 15(3) 15(6) 14(2) 14(3)

1e-3 12 28(3) 27(5) 27(4) 26(3)

1e-1 88 14(1) 14(1) 14(2) 13(2)

1e-3 90 16(2) 15(1) 15(2) 14(2)

1e-5 12 40(5) 39(5) 39(3) 37(2) 1e-5 92 17(2) 16(1) 17(2) 16(2)

1e-7 12 52(5) 51(5) 50(3) 50(4)

#succ 15/15 15/15 15/15 15/15 15/15

∆fopt 1e1 f13 132 BIPOP-a 3.2(2) IPOP-aC 3.0(2) NBIPOP-2.7(1) NIPOP-a 2.9(2)

1e0 195 3.8(1) 4.1(2) 3.3(2) 4.4(2)

1e-1 250 4.3(0.9) 4.2(0.8) 3.9(1) 4.7(1)

1e-3 1310 1.2(0.1) 1.2(0.2) 1.1(0.2) 1.2(0.2)

1e-5 1752 1.2(0.1) 1.2(0.1) 1.2(0.2) 1.3(0.3)

1e-7 2255 1.1(0.2) 1.1(0.1) 1.2(0.1) 1.2(0.2)

#succ 15/15 15/15 15/15 15/15 15/15

1e-7 94 18(1) 18(1) 18(2) 17(2)

#succ 15/15 14/15 15/15 15/15 15/15

∆fopt 1e1 f14 10 BIPOP-a 2.3(3) IPOP-aC 1.5(2) NBIPOP- 1.7(2) NIPOP-a 2.1(2)

1e0 41 2.6(1) 2.2(1) 2.5(1) 3.1(1)

1e-1 58 3.3(0.8) 3.2(0.8) 3.6(0.8) 3.9(1)

1e-3 139 3.9(0.7) 3.6(0.5) 4.1(0.5) 4.3(0.7)

1e-5 251 3.9(0.7) 3.8(0.6) 4.2(0.5) 4.0(0.5)

1e-7 476 3.2(0.3) 2.9(0.3) 3.2(0.4) 3.2(0.4)

#succ 15/15 15/15 15/15 15/15 15/15

#succ 15/15 15/15 4/15 15/15 15/15

∆fopt 1e1 f15 511 BIPOP-a 1.5(2) IPOP-aC 1.5(2) NBIPOP- 1.7(2) NIPOP-a 1.4(2)

1e0 9310 1.1(0.8) 0.89(0.5) 0.99(1) 1.1(0.9)

1e-1 19369 1.4(1) 1.0(0.7) 1.6(0.9) 1.2(0.7)

1e-3 20073 1.3(1) 1.0(0.7) 1.6(0.9) 1.2(0.7)

1e-5 20769 1.3(1) 1.0(0.7) 1.5(0.9) 1.2(0.7)

1e-7 21359 1.3(1) 1.0(0.7) 1.5(0.8) 1.2(0.7)

#succ 14/15 15/15 15/15 15/15 15/15

1e-7 #succ 1903 15/15 ∞ 5e6 0/15 ∞ 9e5 0/15 3387(3087) 9/15 ∞ 5e6 0/15

∆fopt 1e1 f16 120 BIPOP-a 3.6(2) IPOP-aC 3.9(4) NBIPOP- 2.6(2) NIPOP-a 1.8(2)

1e0 612 3.5(3) 2.4(2) 4.6(6) 2.7(5)

1e-1 2662 1.8(2) 1.7(2) 2.4(2) 1.0(1)

1e-3 10449 0.74(1.0) 0.82(0.7) 0.99(1) 0.56(0.6)

1e-5 11644 0.71(0.8) 0.84(0.6) 0.94(1.0) 0.55(0.5)

1e-7 12095 0.71(0.8) 0.85(0.6) 0.93(1.0) 0.57(0.5)

#succ 15/15 15/15 15/15 15/15 15/15

1e-1 1e-3 1e-5 1e-7 1637 1646 1650 1654 190(397) 190(395) 190(394) 190(393) 1359(1774) 1353(1691) 1350(1687) 1348(1708) 473(727) 471(723) 470(721) 470(720) 799(1357) 795(1349) 793(1345) 792(1342)

1e0 1e-1 1633 1688 1.4e4(1e4) ∞ ∞ ∞ 738(708) 3819(3481) 2.2e4(2e4) ∞

1e-3 1817 ∞ ∞ 3547(3233) ∞

1e-5 1886 ∞ ∞ 3418(3116) ∞

∆fopt 1e1 f5 10 BIPOP-a 5.7(3) IPOP-aC 4.6(2) NBIPOP- 4.2(2) NIPOP-a 4.1(2)

1e0 10 6.9(3) 6.3(2) 5.6(2) 5.9(3)

1e-1 10 6.9(3) 6.8(2) 5.9(2) 6.1(3)

1e-3 10 6.9(3) 6.8(2) 5.9(2) 6.1(3)

1e-5 10 6.9(3) 6.8(2) 5.9(2) 6.1(3)

1e-7 10 6.9(3) 6.8(2) 5.9(2) 6.1(3)

#succ 15/15 15/15 15/15 15/15 15/15

∆fopt 1e1 f17 5.2 BIPOP-a 3.8(5) IPOP-aC 4.3(5) NBIPOP- 6.5(6) NIPOP-a 5.5(4)

1e0 215 0.86(0.3) 0.89(0.4) 5.7(7) 1.00(0.4)

1e-1 899 1.1(2) 0.53(0.2) 2.1(2) 0.88(2)

1e-3 3669 0.83(0.5) 0.77(0.5) 1.1(1) 0.98(0.9)

1e-5 6351 0.94(0.7) 1.00(0.5) 1.0(0.6) 0.90(0.4)

1e-7 7934 1.4(0.4) 1.1(0.9) 1.3(0.7) 1.0(0.4)

#succ 15/15 15/15 15/15 15/15 15/15

∆fopt 1e1 f6 114 BIPOP-a 2.0(1) IPOP-aC 2.5(0.8) NBIPOP- 2.1(1) NIPOP-a 2.0(0.8)

1e0 214 1.9(0.8) 2.1(0.6) 1.9(0.6) 1.7(0.5)

1e-1 281 2.2(0.5) 2.2(0.4) 2.0(0.6) 1.9(0.3)

1e-3 580 1.6(0.3) 1.6(0.2) 1.5(0.3) 1.5(0.2)

1e-5 1038 1.2(0.2) 1.2(0.1) 1.1(0.2) 1.2(0.1)

1e-7 1332 1.2(0.1) 1.2(0.1) 1.1(0.2) 1.2(0.1)

#succ 15/15 15/15 15/15 15/15 15/15

∆fopt 1e1 f18 103 BIPOP-a 1.2(0.3) IPOP-aC 3.5(0.8) NBIPOP-1.0(0.6) NIPOP-a 1.1(0.7)

1e0 378 2.0(3) 1.6(0.5) 3.1(7) 1.5(1)

1e-1 3968 0.62(0.5) 0.70(0.5) 0.68(0.9) 0.49(0.6)

1e-3 9280 0.76(0.3) 0.77(0.3) 0.99(0.4) 0.99(0.7)

1e-5 10905 0.86(0.3) 0.80(0.3) 1.0(0.4) 1.1(0.7)

1e-7 12469 0.98(0.3) 0.84(0.3) 1.1(0.4) 1.1(0.4)

#succ 15/15 15/15 15/15 15/15 15/15

∆fopt 1e1 f7 24 BIPOP-a 6.9(4) IPOP-aC 4.0(3) NBIPOP- 4.8(2) NIPOP-a 6.0(3)

1e0 324 1.3(1) 0.87(0.2) 1.3(1) 1.3(1)

1e-1 1171 1.0(0.9) 0.70(0.6) 0.86(0.9) 0.93(0.9)

1e-3 1572 0.93(0.7) 0.69(0.5) 0.86(0.6) 0.99(0.7)

1e-5 1572 0.93(0.7) 0.69(0.5) 0.86(0.6) 0.99(0.7)

1e-7 1597 0.95(0.7) 0.70(0.5) 0.88(0.6) 1.1(0.7)

#succ 15/15 15/15 15/15 15/15 15/15

∆fopt 1e1 f19 1 BIPOP-a 22(14) IPOP-aC 14(10) NBIPOP- 20(20) NIPOP-a 16(16)

1e0 1 1466(1186) 1207(1125) 2026(1762) 1813(1877)

∆fopt 1e1 f8 73 BIPOP-a 3.0(1) IPOP-aC 2.8(1) NBIPOP- 2.9(0.9) NIPOP-a 4.1(3)

1e0 273 5.9(5) 3.0(1) 4.1(4) 4.4(5)

1e-1 336 6.2(4) 3.6(1) 4.6(4) 4.8(4)

1e-3 391 6.2(4) 4.0(1) 4.8(3) 5.0(4)

1e-5 410 6.4(4) 4.2(1) 5.1(3) 5.2(4)

1e-7 422 6.7(4) 4.5(1.0) 5.3(3) 5.5(4)

#succ 15/15 15/15 15/15 15/15 15/15

∆fopt 1e1 f20 16 BIPOP-a 3.4(3) IPOP-aC 3.9(2) NBIPOP- 3.6(3) NIPOP-a 3.2(2)

1e0 851 10(11) 10(4) 11(12) 10(5)

∆fopt 1e1 f9 35 BIPOP-a 7.9(4) IPOP-aC 5.4(1) NBIPOP-5.2(1) NIPOP-a 5.6(1)

1e0 127 10(10) 6.2(2) 6.2(3) 5.4(2)

1e-3 300 6.6(5) 5.0(1.0) 5.0(0.9) 4.6(0.8)

1e-5 335 6.4(4) 5.0(0.8) 5.0(1.0) 4.6(0.8)

1e-7 369 6.3(4) 4.9(0.8) 5.0(0.9) 4.6(0.8)

#succ 15/15 15/15 15/15 15/15 15/15

1e0 ∆fopt 1e1 f21 41 1157 BIPOP-a 1.8(0.9) 7.4(8) 7.3(8) IPOP-aC 3.5(1) 11(10) NBIPOP- 2.1(2) NIPOP-a 4.1(1) 76(145)

∆fopt 1e1 f10 349 BIPOP-a 2.6(0.9) IPOP-aC 2.5(0.8) NBIPOP- 2.8(0.8) NIPOP-a 2.7(0.7)

1e0 500 2.2(0.6) 2.2(0.3) 2.2(0.5) 2.3(0.3)

1e-1 574 2.1(0.3) 2.1(0.3) 2.1(0.2) 2.1(0.3)

1e-3 626 2.2(0.3) 2.2(0.3) 2.2(0.2) 2.2(0.3)

1e-5 829 1.9(0.1) 1.8(0.2) 1.8(0.2) 1.8(0.2)

1e-7 880 2.0(0.1) 1.9(0.2) 1.9(0.2) 1.9(0.2)

#succ 15/15 15/15 15/15 15/15 15/15

∆fopt 1e1 f22 71 BIPOP-a 14(13) IPOP-aC 8.8(10) NBIPOP- 8.9(11) NIPOP-a 4.0(10)

1e0 386 16(22) 21(26) 14(18) 258(468)

1e-1 938 38(92) 65(74) 18(19) 338(715)

1e-3 1008 36(86) 270(374) 17(18) 316(665)

∆fopt 1e1 f11 143 BIPOP-a 5.1(2) IPOP-aC 5.6(0.6) NBIPOP- 6.1(1) NIPOP-a 5.2(1)

1e0 202 4.6(1) 4.7(0.5) 5.0(0.8) 4.4(0.7)

1e-1 763 1.4(0.2) 1.4(0.1) 1.5(0.2) 1.3(0.1)

1e-3 1177 1.1(0.2) 1.0(0.1) 1.1(0.1) 1.0(0.1)

1e-5 1467 0.94(0.1) 0.95(0.1) 0.97(0.1) 0.93(0.1)

1e-7 1673 0.91(0.1) 0.92(0.1) 0.94(0.1) 0.90(0.1)

#succ 15/15 15/15 15/15 15/15 15/15

∆fopt 1e1 f23 3.0 BIPOP-a 3.2(3) IPOP-aC 1.6(1) NBIPOP-1.6(2) NIPOP-a 3.2(2)

1e0 518 11(9) 20(19) 8.2(8) 15(21)

1e-1 14249 1.6(1) 76(124) 1.4(2) 1.8(1)

1e-3 1e-5 1e-7 31654 33030 34256 0.86(0.6) 0.85(0.6) 0.84(0.6) 34(56) 33(38) 32(51) 0.97(1) 0.95(1) 1.1(0.9) 0.86(0.7) 0.86(0.7) 0.87(0.7)

∆fopt 1e1 f12 108 BIPOP-a 12(9) IPOP-aC 8.8(7) NBIPOP-10(6) NIPOP-a 11(16)

1e0 268 8.4(6) 5.9(7) 6.9(6) 8.2(11)

1e-1 371 7.7(6) 5.7(5) 6.5(6) 8.0(9)

1e-3 461 7.3(6) 6.0(5) 6.5(5) 7.9(8)

1e-5 1303 3.0(3) 2.6(2) 2.7(2) 3.4(3)

1e-7 1494 3.2(3) 2.6(2) 2.8(2) 3.4(3)

#succ 15/15 15/15 15/15 15/15 15/15

∆fopt 1e1 f24 1622 BIPOP-a 1.3(1) IPOP-aC 2.6(2) NBIPOP- 2.0(1) NIPOP-a 1.8(1)

1e-1 214 7.7(6) 5.7(1) 5.6(2) 5.0(1)

1e-1 242 186(177) 123(152) 156(138) 324(473) 1e-1 38111 2.4(1) 1.4(2) 2.4(1) 2.1(2)

1e-1 1674 49(23) 32(36) 31(62) 272(620)

1e0 2.2e5 1.0(1) 41(50) 0.64(0.5) 2.1(4)

1e-3 1.2e5 2.0(1) 0.95(0.7) 2.6(4) 2.7(3) 1e-3 54470 1.7(0.9) 1.1(1) 1.8(1) 1.6(1)

1e-3 1705 50(22) 33(40) 30(61) 269(609)

1e-1 6.4e6 1.1(1) ∞ 0.92(1) 0.61(0.7)

1e-3 9.6e6 1.3(1) ∞ 0.81(1.0) 0.46(0.4)

1e-5 1.2e5 2.0(1) 0.96(0.7) 2.6(4) 2.7(3)

1e-5 54861 1.8(0.9) 1.1(1) 1.8(1) 1.6(1) 1e-5 1729 50(22) 33(41) 30(60) 266(601) 1e-5 1040 35(84) 262(364) 17(18) 307(644)

1e-5 1.3e7 0.96(1) ∞ 0.71(0.9) 0.35(0.3)

1e-7 1.2e5 2.0(1) 0.96(0.7) 2.6(4) 2.7(3)

#succ 15/15 15/15 15/15 15/15 15/15

1e-7 55313 1.8(0.9) 1.1(1) 1.8(1) 1.7(1)

#succ 14/15 15/15 15/15 15/15 15/15

1e-7 1757 49(22) 33(41) 29(59) 263(591)

#succ 14/15 15/15 14/15 15/15 15/15

1e-7 1068 34(82) 257(356) 17(17) 300(628)

#succ 14/15 15/15 9/15 15/15 15/15

1e-7 1.3e7 0.96(1) ∞ 1e6 0.88(1) 0.35(0.3)

#succ 15/15 15/15 8/15 15/15 15/15 #succ 3/15 5/15 0/15 5/15 11/15

Table 1: Expected running time (ERT in number of function evaluations) divided by the respective best ERT measured during BBOB-2009 (given in the respective first row) for different ∆f values in dimension 5. The central 80% range divided by two is given in braces. The median number of conducted function evaluations is additionally given in italics, if ERT(10−7 ) = ∞. #succ is the number of trials that reached the final target fopt + 10−8 . Best results are printed in bold.

∆fopt 1e1 f1 83 BIPOP-a 9.4(1) IPOP-aC 9.3(1) NBIPOP- 9.5(1) NIPOP-a 10(0.8)

1e0 83 15(2) 15(1) 15(1) 15(1)

1e-1 83 21(2) 21(1) 22(1) 21(1.0)

1e-3 83 33(2) 33(2) 34(0.9) 34(1)

1e-5 83 45(3) 45(2) 46(2) 46(2)

1e-7 83 58(2) 57(2) 58(1) 58(1)

#succ 15/15 15/15 8/8 15/15 15/15

∆fopt 1e1 f13 2029 BIPOP-a 2.0(0.2) IPOP-aC 1.6(0.4) NBIPOP- 2.5(3) NIPOP-a 2.4(3)

1e0 6916 3.8(3) 1.8(1) 3.2(2) 2.4(3)

1e-1 8734 5.3(4) 5.6(4) 5.0(4) 4.1(4)

1e-3 71936 1.3(0.9) 1.4(1) 1.2(0.9) 1.4(1)

1e-5 98467 1.6(1) 1.4(0.8) 2.0(2) 1.6(0.8)

1e-7 1.2e5 2.0(1) 1.9(0.9) 2.8(2) 1.7(0.8)

∆fopt 1e1 f2 796 BIPOP-a 38(3) IPOP-aC 37(3) NBIPOP- 37(3) NIPOP-a 37(4)

1e0 797 45(3) 43(5) 43(4) 43(4)

1e-1 799 49(3) 48(4) 47(5) 48(4)

1e-3 800 55(3) 55(3) 53(4) 53(3)

1e-5 802 57(2) 57(3) 57(2) 57(2)

1e-7 804 59(2) 58(3) 59(2) 58(1)

#succ 15/15 15/15 8/8 15/15 15/15

∆fopt 1e1 f14 304 BIPOP-a 2.5(0.2) IPOP-aC 2.5(0.5) NBIPOP- 2.5(0.5) NIPOP-a 2.5(0.4)

1e0 616 2.3(0.3) 2.3(0.2) 2.4(0.2) 2.3(0.3)

1e-1 777 2.9(0.2) 2.9(0.3) 3.0(0.2) 3.0(0.3)

1e-3 2207 3.5(0.2) 3.4(0.2) 3.5(0.3) 3.4(0.1)

1e-5 4825 3.9(0.2) 3.9(0.2) 3.9(0.2) 3.8(0.2)

1e-7 #succ 57711 15/15 0.59(0.0)↓4 15/15 0.57(0.0)↓2 8/8 0.59(0.0)↓4 15/15 0.60(0.0)↓4 15/15

∆fopt 1e1 f15 1.9e5 BIPOP-a 1.2(0.5) IPOP-aC 0.72(0.3) NBIPOP- 1.0(0.4) NIPOP-a 0.92(0.3)

1e0 1e-1 7.9e5 1.0e6 1.1(0.5) 1.1(0.4) 0.43(0.1)↓20.60(0.4) 0.71(0.3)↓2 0.75(0.3) 0.61(0.2)↓ 0.55(0.2)

1e-3 1.1e6 1.1(0.4) 0.61(0.4) 0.76(0.3) 0.56(0.2)

1e-5 1.1e6 1.1(0.4) 0.62(0.5) 0.77(0.3) 0.57(0.2)

1e-7 1.1e6 1.1(0.4) 0.63(0.5) 0.77(0.3) 0.58(0.2)

∆fopt 1e1 f16 5244 BIPOP-a 1.3(0.4) IPOP-aC 0.91(0.3) NBIPOP- 0.97(0.3) NIPOP-a 1.2(0.4)

1e0 72122 0.96(0.3) 1.1(0.5) 0.78(0.4) 0.65(0.2)

1e-1 1e-3 1e-5 1e-7 #succ 3.2e5 1.4e6 2.0e6 2.0e6 15/15 0.80(0.4) 0.54(0.3) 0.50(0.3) 0.51(0.3) 15/15 1.0(0.9) 0.51(0.7) 1.4(1) 1.4(1) 8/8 0.34(0.1)↓3 0.38(0.3)↓2 0.46(0.4) 0.74(1) 15/15 ⋆ ⋆ 0.23(0.1)↓40.21(0.2)↓30.16(0.1)↓30.18(0.1)↓315/15

∆fopt 1e1 f17 399 BIPOP-a 1.1(0.3) IPOP-aC 1.0(0.4) NBIPOP- 1.0(0.4) NIPOP-a 0.97(0.3)

1e0 4220 0.64(0.2) 0.52(0.2) 0.57(0.2) 0.52(0.1)

1e-1 14158 1.6(1) 1.3(1) 1.2(1) 0.97(1)

1e-3 51958 1.1(0.4) 1.3(0.9) 1.2(0.5) 1.00(0.4)

1e-5 1.3e5 1.4(1) 0.97(0.2) 1.0(0.3) 1.1(0.6)

1e-7 2.7e5 0.87(0.4) 0.83(0.3) 0.81(0.3) 0.70(0.2)↓

#succ 14/15 15/15 8/8 15/15 15/15

∆fopt 1e1 f18 1442 BIPOP-a 0.94(0.2) IPOP-aC 0.96(0.4) NBIPOP- 1.0(0.2) NIPOP-a 0.95(0.2)

1e0 16998 0.51(0.8) 0.68(0.9) 0.97(1) 0.58(0.8)

1e-1 47068 1.0(0.4) 1.0(0.4) 1.1(0.6) 0.75(0.1)

1e-3 1.9e5 0.98(0.4) 0.66(0.2)↓ 0.93(0.4) 0.71(0.2)↓

1e-5 6.7e5 0.88(0.7) 0.45(0.4) 0.57(0.4) 0.50(0.3)

1e-7 9.5e5 0.67(0.5) 0.48(0.2) 0.53(0.3) 0.42(0.2)

#succ 15/15 15/15 8/8 15/15 15/15

∆fopt 1e1 f19 1 BIPOP-a 396(82) IPOP-aC 462(122) NBIPOP- 424(90) NIPOP-a 436(102)

1e0 1e-1 1e-3 1e-5 1e-7 #succ 1 1.4e6 2.6e7 4.5e7 4.5e7 8/15 6.7e4(5e4)0.87(0.7) 1.2(1) 1.0(0.9) 1.0(1.0) 9/15 4.4e4(2e4)0.57(0.5) 0.34(0.1)↓20.20(0.1)↓20.20(0.1)↓2 8/8 8.3e4(6e4)0.97(0.6) 0.81(0.5) 1.1(0.9) 1.1(0.9) 9/15 8.2e4(4e4)1.9(6) 0.48(0.3)↓ 0.32(0.2)↓ 0.32(0.2)↓ 15/15 1e0 1e-1 1e-3 1e-5 1e-7 #succ 1.3e5 1.6e8 ∞ ∞ ∞ 0 9.0(4) 0.34(0.4) . . . 0/15 8.1(5) 0.18(0.2) . . . 0/8 8.5(3) 0.39(0.4) . . . 0/15 6.5(2) 0.32(0.3) . . . 0/15

∆fopt 1e1 f3 15526 BIPOP-a 2395(2759) IPOP-aC ∞ NBIPOP- 8177(9018) NIPOP-a 4615(5541) ∆fopt f4 BIPOP-a IPOP-aC NBIPOPNIPOP-a

1e1 15536 ∞ ∞ ∞ ∞

1e0 15602 ∞ ∞ ∞ ∞

1e-1 15612 ∞ ∞ ∞ ∞

1e-3 15646 ∞ ∞ ∞ ∞

1e-5 15651 ∞ ∞ ∞ ∞

1e-7 15656 ∞ 4e7 ∞ 6e6 ∞ 4e7 ∞ 4e7

#succ 15/15 0/15 0/8 0/15 0/15

1e0 15601 ∞ ∞ ∞ ∞

1e-1 15659 ∞ ∞ ∞ ∞

1e-3 15703 ∞ ∞ ∞ ∞

1e-5 15733 ∞ ∞ ∞ ∞

1e-7 2.8e5 ∞ 4e7 ∞ 6e6 ∞ 4e7 ∞ 4e7

#succ 6/15 0/15 0/8 0/15 0/15

∆fopt 1e1 f5 98 BIPOP-a 4.6(0.7) IPOP-aC 4.8(0.5) NBIPOP-4.5(0.9) NIPOP-a 4.8(0.7)

1e0 116 4.5(0.8) 4.7(0.6) 4.5(0.8) 4.6(0.8)

1e-1 120 4.4(0.7) 4.5(0.7) 4.4(0.7) 4.5(0.8)

1e-3 121 4.4(0.7) 4.5(0.7) 4.4(0.7) 4.5(0.8)

1e-5 121 4.4(0.7) 4.5(0.7) 4.4(0.7) 4.5(0.8)

1e-7 121 4.4(0.7) 4.5(0.7) 4.4(0.7) 4.5(0.8)

#succ 15/15 15/15 8/8 15/15 15/15

∆fopt 1e1 f6 3507 BIPOP-a 1.6(0.2) IPOP-aC 1.5(0.1) NBIPOP- 1.5(0.2) NIPOP-a 1.6(0.3)

1e0 5523 1.5(0.3) 1.4(0.2) 1.4(0.2) 1.4(0.2)

1e-1 7168 1.4(0.2) 1.4(0.2) 1.3(0.1) 1.4(0.1)

1e-3 11538 1.3(0.2) 1.3(0.2) 1.2(0.1) 1.3(0.1)

1e-5 15007 1.3(0.1) 1.3(0.2) 1.2(0.1) 1.3(0.1)

1e-7 19222 1.3(0.1) 1.3(0.1) 1.2(0.1) 1.2(0.1)

#succ 15/15 15/15 8/8 15/15 15/15

∆fopt 1e1 f7 10698 BIPOP-a 1.2(0.9) IPOP-aC 1.1(0.8) NBIPOP- 1.2(0.9) NIPOP-a 0.89(0.8)

1e0 17839 4.5(2) 2.5(0.4)⋆ 3.2(0.7) 3.2(1.0)

1e-1 41037 2.4(0.9) 1.3(0.4) 1.8(0.6) 1.9(0.5)

1e-3 66294 1.5(0.6) 0.86(0.3) 1.2(0.4) 1.2(0.3)

1e-5 66294 1.5(0.6) 0.86(0.3) 1.2(0.4) 1.2(0.3)

1e-7 68145 1.5(0.6) 0.84(0.3) 1.1(0.4) 1.2(0.3)

#succ 15/15 15/15 8/8 15/15 15/15

∆fopt 1e1 f8 7080 BIPOP-a 5.5(0.6) IPOP-aC 5.4(0.3) NBIPOP- 5.5(0.7) NIPOP-a 5.5(0.4)

1e0 10655 6.1(4) 4.7(0.2) 6.5(4) 7.8(8)

1e-1 11012 6.3(4) 4.9(0.2) 6.6(4) 7.9(8)

1e-3 11430 6.3(4) 4.9(0.2) 6.6(3) 7.8(8)

1e-5 11701 6.2(3) 4.9(0.1) 6.6(3) 7.8(8)

1e-7 11969 6.2(3) 4.9(0.1) 6.6(3) 7.8(8)

#succ 15/15 15/15 8/8 15/15 15/15

∆fopt 1e1 f9 6122 BIPOP-a 6.0(0.8) IPOP-aC 6.3(0.7) NBIPOP- 6.3(0.7) NIPOP-a 6.3(0.8)

1e0 12982 4.5(3) 4.6(3) 4.6(3) 5.0(3)

1e-1 13300 4.6(3) 4.7(3) 4.8(3) 5.1(3)

1e-3 13651 4.7(3) 4.8(3) 4.8(3) 5.2(3)

1e-5 13909 4.7(3) 4.8(3) 4.8(3) 5.2(3)

1e-7 14142 4.7(3) 4.8(3) 4.9(3) 5.2(3)

#succ 15/15 15/15 8/8 15/15 15/15

∆fopt 1e1 f10 25890 BIPOP-a 1.2(0.1) IPOP-aC 1.2(0.2) NBIPOP-1.1(0.1) NIPOP-a 1.1(0.1)

1e0 30368 1.1(0.1) 1.1(0.1) 1.1(0.1) 1.1(0.1)

1e-1 36796 1.0(0.1) 1.1(0.1) 1.0(0.1) 1.0(0.1)

1e-3 1e-5 1e-7 #succ 56007 65128 70824 15/15 0.77(0.0)↓40.70(0.0)↓40.66(0.0)↓415/15 0.78(0.0)↓2 0.71(0.0)↓2 0.67(0.0)↓2 8/8 0.77(0.0)↓4 0.71(0.0)↓4 0.67(0.0)↓4 15/15 0.77(0.0)↓40.70(0.0)↓4 0.67(0.0)↓4 15/15

∆fopt 1e1 f11 2368 BIPOP-a 5.0(0.3) IPOP-aC 5.0(0.3) NBIPOP- 5.0(0.2) NIPOP-a 5.0(0.2)

1e0 4855 2.6(0.1) 2.6(0.1) 2.7(0.1) 2.7(0.1)

1e-1 11681 1.2(0.0) 1.2(0.0) 1.2(0.0) 1.2(0.0)

1e-3 1e-5 1e-7 #succ 29749 38949 48211 15/15 0.51(0.0)↓40.42(0.0)↓40.37(1e-2)↓4 15/15 0.51(0.0)↓2 0.42(7e-3)↓2 0.37(5e-3)↓2 8/8 0.51(0.0)↓4 0.43(8e-3)↓4 0.37(7e-3)↓4 15/15 0.51(9e-3)↓4 0.42(9e-3)↓4 0.37(9e-3)↓4 15/15

∆fopt 1e1 f12 4169 BIPOP-a 1.9(1) IPOP-aC 1.2(0.8) NBIPOP- 2.3(1) NIPOP-a 1.9(2)

1e0 7452 1.9(1) 1.3(1.0) 2.2(1) 2.0(1)

1e-1 9174 2.2(1) 1.7(1) 2.4(1.0) 2.2(1)

1e-3 13146 2.1(0.7) 1.8(0.4) 2.2(0.7) 2.1(0.7)

1e-5 22758 1.5(0.4) 1.3(0.3) 1.5(0.4) 1.5(0.4)

1e-7 25192 1.5(0.4) 1.3(0.3) 1.5(0.4) 1.5(0.4)

#succ 15/15 15/15 8/8 15/15 15/15

∆fopt 1e1 f20 222 BIPOP-a 4.0(0.4) IPOP-aC 3.9(0.8) NBIPOP- 4.0(0.8) NIPOP-a 4.0(0.6) ∆fopt 1e1 f21 1044 BIPOP-a 7.5(11) IPOP-aC 7.1(11) NBIPOP- 4.9(6) NIPOP-a 14(22)

1e0 21144 60(19) 421(491) 10(20) 440(890)

1e-1 1.0e5 37(56) ∞ 5.1(8) 173(228)

∆fopt 1e1 f22 3090 BIPOP-a 12(20) IPOP-aC 144(492) NBIPOP- 12(6) NIPOP-a 179(468)

1e0 35442 343(565) 93(127) 112(120) 583(914)

1e-1 6.5e5 201(223) ∞ 32(41) ∞

∆fopt 1e1 f23 7.1 BIPOP-a 8.4(9) IPOP-aC 9.2(13) NBIPOP- 8.6(11) NIPOP-a 5.9(7)

1e0 11925 7.8(7) ∞ 10(12) 61(18)

1e-1 75453 1.3(1) ∞ 1.6(2) 11(3)

∆fopt 1e1 f24 5.8e6 BIPOP-a 3.6(4) IPOP-aC ∞ NBIPOP- 2.1(3) NIPOP-a 1.2(1)

1e0 9.8e7 1.4(1) ∞ 0.19(0.2) 0.15(0.2)

1e-1 3.0e8 ∞ ∞ 0.97(1) 0.44(0.5)

1e-3 1.0e5 37(56) ∞ 5.1(8) 172(227) 1e-3 6.5e5 200(222) ∞ 32(39) ∞

#succ 15/15 15/15 8/8 15/15 15/15

1e-5 1.0e5 37(56) ∞ 5.1(8) 171(226)

1e-7 1.0e5 37(55) ∞ 3e6 5.1(8) 171(201)

#succ 26/30 15/15 0/8 15/15 12/15

1e-5 6.5e5 200(201) ∞ 32(40) ∞

1e-7 6.5e5 199(214) ∞ 3e6 32(40) ∞ 4e7

#succ 8/30 4/15 0/8 12/15 0/15

1e-3 1e-5 1e-7 1.3e6 3.2e6 3.4e6 1.9(1) 1.00(0.4) 0.99(0.4) ∞ ∞ ∞ 4e6 1.3(0.4) 0.58(0.2) 0.59(0.2) 0.72(0.2) 0.36(0.2)⋆ 0.38(0.2)⋆ 1e-3 3.0e8 ∞ ∞ 0.97(1.0) 0.44(0.5)

#succ 15/15 15/15 8/8 15/15 15/15

1e-5 3.0e8 ∞ ∞ 0.97(1) 0.44(0.5)

1e-7 3.0e8 ∞ 4e7 ∞ 1e7 0.97(1.0) 0.44(0.5)

#succ 15/15 15/15 0/8 15/15 15/15 #succ 1/15 0/15 0/8 2/15 4/15

Table 2: Expected running time (ERT in number of function evaluations) divided by the respective best ERT measured during BBOB-2009 (given in the respective first row) for different ∆f values in dimension 40. The central 80% range divided by two is given in braces. The median number of conducted function evaluations is additionally given in italics, if ERT(10−7 ) = ∞. #succ is the number of trials that reached the final target fopt + 10−8 . Best results are printed in bold.

Black-box optimization benchmarking of NIPOP-aCMA ...

Jul 11, 2012 - 1 Sphere. BIPOP-aCMA. IPOP-aCMA. NBIPOP-aCMA. NIPOP-aCMA. 2 3. 5. 10. 20. 40. 0. 1. 2. 3. 4 ftarget=1e-08. 2 Ellipsoid separable. 2 3. 5.

758KB Sizes 0 Downloads 392 Views

Recommend Documents

Modeling, Optimization and Performance Benchmarking of ...
Modeling, Optimization and Performance Benchmarking of Multilayer (1).pdf. Modeling, Optimization and Performance Benchmarking of Multilayer (1).pdf. Open.

Benchmarking the benchmarking models
decades and its significance as a practical method in developing critical areas of business is indisputable. It can be said as a management tool for attaining or exceeding the performance goals by learning from best practices and understanding the pr

Benchmarking across Borders: Electoral ... - University of Rochester
Aug 3, 2012 - series dataset without cross-national benchmarking, the higher growth ..... should lead to massive electoral turnover, as incum- bents are punished for ...... mining their “correct” vote, and these heuristics may allow them to ...

Resource Availability Based Performance Benchmarking of Virtual ...
Download. Connect more apps... Try one of the apps below to open or edit this item. Resource Availability Based Performance Benchmarking of Virtual Machine Migrations.pdf. Resource Availability Based Performance Benchmarking of ...

Client-centric benchmarking of eventual consistency for cloud storage ...
Client-centric benchmarking of eventual consistency for cloud storage systems. Wojciech Golab1, Muntasir Raihan Rahman2, Alvin AuYoung3,. Kimberly Keeton3, Jay J. ... J. López, G. Gibson, A. Fuchs, and B. Rinaldi. YCSB++: benchmarking and performanc

ACDC-JS: Explorative Benchmarking of ... - Research at Google
Oct 20, 2014 - information on garbage collection latency and compiler latency, respectively ..... All experiments ran on a desktop machine with an Intel i5-2400.

load testing, benchmarking, and application ...
Note that each file's download starts ..... servers, database management systems, ERP systems, transaction .... type of problem (e.g., DNS server vs. the load.

pdf-106\portfolio-performance-measurement-and-benchmarking ...
Page 1 of 11. PORTFOLIO PERFORMANCE. MEASUREMENT AND BENCHMARKING. (MCGRAW-HILL FINANCE & INVESTING) BY. JON A. CHRISTOPHERSON, DAVID R. CARINO, WAYNE E. DOWNLOAD EBOOK : PORTFOLIO PERFORMANCE MEASUREMENT AND. BENCHMARKING (MCGRAW-HILL FINANCE ...

Towards the Knowledge-Driven Benchmarking of ...
originally emerged to support the operation of complex ... Autonomic systems aim to deliver this cost reduction ... Cost of extra resources needed to support the.

Towards the Knowledge-Driven Benchmarking of ...
relies on an SLA, and will tend to be highly application specific. Ideally ..... Semantic Web Conference (ISWC2003), October 20-23,. 2003, Sanibel Island, Florida ...

Cheap Singapore Cable Tv Box Doole Blackbox C801 Plus +Wifi ...
Cheap Singapore Cable Tv Box Doole Blackbox C801 Pl ... , Football Game Free Shipping & Wholesale Price.pdf. Cheap Singapore Cable Tv Box Doole ...

white box testing and blackbox testing pdf
There was a problem previewing this document. Retrying... Download. Connect more apps... Try one of the apps below to open or edit this item. white box testing ...

Benchmarking Linear Logic Translations
Ideas from linear logic have been influential in fields such as programming languages, ..... theorems, we generated 244 different ILL sequents using 4 automatic ...

Benchmarking Women's Leadership - Colorado Women's College
Aug 18, 2013 - Business and Commercial Banking . .... While fewer in number in the 21st century, wom- en's colleges ..... number of women students and ...... $1800. (BLS 2012b). 2008. 2011. Median Weekly Earnings of Educators by Year.

Benchmarking the Compiler Vectorization for Multimedia Applications
efficient way to exploit the data parallelism hidden in ap- plications. ... The encoder also employs intra-frame analysis when cost effective. ... bigger set of data.

Benchmarking and Evaluating Unified Memoryfor ...
machine learning. Therefore, GPUs will remain a crucial component in supercomputing systems in the foreseeable future. For instance, the next generation supercomputer in. ORNL, Summit, will ... result, CPU and GPU could not access each other's memory

Benchmarking Women's Leadership - Colorado Women's College
Aug 18, 2013 - sit in leadership positions in the top ten organiza- ... technology and social media, where gatekeepers ...... that campaigns with any women.

Benchmarking River Water Quality in Malaysia
The water quality status of rivers in. Malaysia has always been a cause for concern for various local authorities, government agencies as well as the public at large. Rivers in Malaysia are generally considered to be polluted with coherent examples s

Cheap 1Pcs V9 Pro Starhub Singapore Blackbox Tv Receiver From ...
Cheap 1Pcs V9 Pro Starhub Singapore Blackbox Tv Re ... x Hd C608 C801 Free Shipping & Wholesale Price.pdf. Cheap 1Pcs V9 Pro Starhub Singapore ...

OPTIMIZATION OF INTENSITY-MODULATED RADIOTHERAPY ...
NTCPs based on EUD formalism with corresponding ob- Fig. 1. (a) Sample DVH used for EUD calculation. (b) EUD for the. DVH in (a) as a function of parameter a. Tumors generally have. large negative values of a, whereas critical element normal struc- t

OPTIMIZATION OF INTENSITY-MODULATED RADIOTHERAPY ...
deviates from EUD0. For a tumor, the subscore attains a. small value when the equivalent uniform dose falls sig- nificantly below EUD0. Similarly, for a normal ...

Blackbox Kriging: Spatial Prediction Without Specifying ...
models. We then use a flexible piecewise-planar variogrIm model as a step in kriging the two-dimensional Wolfamp Aquifer data, v.ilhoul the need to assume that .... SPATIAL PREDICTION resulting variograms will be valid. In fact, the valid variogram f

white box testing and blackbox testing pdf
testing pdf. Download now. Click here if your download doesn't start automatically. Page 1 of 1. white box testing and blackbox testing pdf. white box testing and ...

CP2K PBE0 benchmarking for ionic crystals - PDFKUL.COM
CP2K PBE0 benchmarking for ionic crystals. Xiaoming Wang. Department of Physics and Astronomy. The University of Toledo [email protected] ...