R Programming/Optimization

• optimize() is devoted to one dimensional optimization problem.
• optim(), nlm(), ucminf() (ucminf) can be used for multidimensional optimization problems.
• nlminb() for constrained optimization.
• quadprog, minqa, rgenoud, trust packages
• Some work is done to improve optimization in R. See Updating and improving optim(), Use R 2009 slides, the R-forge optimizer page and the corresponding packages including optimx.

Numerical Methods

One dimensional problem

The one dimensional problem :

> func <- function(x){
+       return ( (x-2)^2 )
+       }
> (func(-2))
[1] 16
>
> # plot your function using the 'curve function'
> curve(func,-4,8) 
>
> # Here is another way to plot the function
> # using a grid
> grid <- seq(-10,10,by=.1) 
> func(grid)
> plot(grid,func(grid))
> 
> # you can find the minimum using the optimize function
> optimize(f=func,interval=c(-10,10))
$minimum
[1] 2
 
$objective
[1] 0

Newton-Raphson

• nlm() provides a Newton algorithm.
• maxLik package for maximization of a likelihood function. This package includes the Newton Raphson method.
• newtonraphson() in the spuRs package.

BFGS

• The BFGS method

> func <- function(x){
+       out <- (x[1]-2)^2 + (x[2]-1)^2
+       return <- out
+       }> 
> optim(par=c(0,0), fn=func, gr = NULL,
+       method = c("BFGS"),
+       lower = -Inf, upper = Inf,
+       control = list(), hessian = T)
> optim(par=c(0,0), fn=func, gr = NULL,
+       method = c("L-BFGS-B"),
+       lower = -Inf, upper = Inf,
+       control = list(), hessian = T)

Conjugate gradient method

• optim() with method="cg".

Trust Region Method

• "trust" package for trust region method

The Nelder-Mead simplex method

• The Nelder Mead Method

> func <- function(x){
+       out <- (x[1]-2)^2 + (x[2]-1)^2
+       return <- out
+       }
> 
> optim(par=c(0,0), fn=func, gr = NULL,
+       method = c("Nelder-Mead"),
+       lower = -Inf, upper = Inf,
+       control = list(), hessian = T)

• The boot package includes another simplex method

Simulation methods

Simulated Annealing

• The Simulated Annealing is an algorithm which is useful to maximise non-smooth functions. It is pre implemented in optim().

> func <- function(x){
+       out <- (x[1]-2)^2 + (x[2]-1)^2
+       return <- out
+       }> 
> optim(par=c(0,0), fn=func, gr = NULL,
+       method = c("SANN"),
+       lower = -Inf, upper = Inf,
+       control = list(), hessian = T)

EM Algorithm

Genetic Algorithm

• rgenoud package for genetic algorithm
• gaoptim package for genetic algorithm

References

• Venables and Ripley, Chapter 16.
• Cameron and Trivedi, Microeconometrics, chapter 10
• Braun and Murdoch (Chapter 7) is a very good reference on optimization using R.