# First steps into Bayesian optimization¶

Ivo Couckuyt, Joachim van der Herten

## Introduction¶

Bayesian optimization is particularly useful for expensive optimization problems. This includes optimization problems where the objective (and constraints) are time-consuming to evaluate: measurements, engineering simulations, hyperparameter optimization of deep learning models, etc. Another area where Bayesian optimization may provide a benefit is in the presence of (a lot of) noise. If your problem does not satisfy these requirements other optimization algorithms might be better suited.

To setup a Bayesian optimization scheme with GPflowOpt you have to:

• define your objective and specify the optimization domain
• setup a GPflow model and choose an acquisition function
• create a BayesianOptimizer

## Objective function¶

In :

import numpy as np
from gpflowopt.domain import ContinuousParameter

def branin(x):
x = np.atleast_2d(x)
x1 = x[:, 0]
x2 = x[:, 1]
a = 1.
b = 5.1 / (4. * np.pi ** 2)
c = 5. / np.pi
r = 6.
s = 10.
t = 1. / (8. * np.pi)
ret = a * (x2 - b * x1 ** 2 + c * x1 - r) ** 2 + s * (1 - t) * np.cos(x1) + s
return ret[:, None]

domain = ContinuousParameter('x1', -5, 10) + \
ContinuousParameter('x2', 0, 15)
domain

Out:

 Name Type Values x1 Continuous [-5. 10.] x2 Continuous [ 0. 15.]

## Bayesian optimizer¶

In :

import gpflow
from gpflowopt.bo import BayesianOptimizer
from gpflowopt.design import LatinHyperCube
from gpflowopt.acquisition import ExpectedImprovement
from gpflowopt.optim import SciPyOptimizer, StagedOptimizer, MCOptimizer

# Use standard Gaussian process Regression
lhd = LatinHyperCube(21, domain)
X = lhd.generate()
Y = branin(X)
model = gpflow.gpr.GPR(X, Y, gpflow.kernels.Matern52(2, ARD=True))
model.kern.lengthscales.transform = gpflow.transforms.Log1pe(1e-3)

# Now create the Bayesian Optimizer
alpha = ExpectedImprovement(model)

acquisition_opt = StagedOptimizer([MCOptimizer(domain, 200),
SciPyOptimizer(domain)])

optimizer = BayesianOptimizer(domain, alpha, optimizer=acquisition_opt, verbose=True)

# Run the Bayesian optimization
r = optimizer.optimize(branin, n_iter=10)
print(r)

iter #  0 - MLL [-13.1] - fmin [4.42]
iter #  1 - MLL [-13.4] - fmin [4.42]
iter #  2 - MLL [-10.6] - fmin [0.723]
iter #  3 - MLL [-9.09] - fmin [0.486]
iter #  4 - MLL [-7.01] - fmin [0.486]
iter #  5 - MLL [-2.69] - fmin [0.446]
iter #  6 - MLL [1.96] - fmin [0.446]
iter #  7 - MLL [4.6] - fmin [0.446]
iter #  8 - MLL [7.37] - fmin [0.4]
iter #  9 - MLL [12.6] - fmin [0.4]
constraints: array([], dtype=float64)
fun: array([0.39970302])
message: 'OK'
nfev: 10
success: True
x: array([[9.40798299, 2.43938799]])


That’s all! Your objective function has now been optimized for 10 iterations.