Additional documentation for the optim package for Octave.

Copyright © Olaf Till <i7tiol@t-online.de>

You can redistribute this documentation and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.

This documentation is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this documentation; if not, see <http://www.gnu.org/licenses/>.

     [file macros.texi not included]

Additional documentation for the optim package for Octave
*********************************************************

This documentation applies to version of the optim package.

The optim package is a collection of additional functions related to numerical optimization.  For Octaves core optimization functions (not contained in this package) Optimization (octave).

Functions for optimization of a scalar objective function
*********************************************************

Frontend nonlin_min for scalar non-linear optimization
======================================================

(nonlin_min)

Settings
--------

The fields of the SETTINGS structure can be set with optimset (octave).

For settings common to all frontends (including these for statistics) Common frontend options.

For additional settings common to all optimization frontends Common optimization options.

Specific defaults:
..................

Algorithm:
"lm_feasible"

Additional settings:
....................

objf_grad
     Function computing the gradient of the objective function with respect to the parameters.  Default: real finite differences.  Will be called with the column vector of parameters and, if it accepts it, an informational structure as arguments.  If objf_grad was specified by the user, the informational structure has the fields f: value of objective function for current parameters, fixed: logical vector indicating which parameters are not optimized, so these partial derivatives need not be computed and can be set to zero, diffp, diff_onesided, lbound, ubound: identical to the user settings of this name, plabels: 1-dimensional cell-array of column-cell-arrays, each column with labels for all parameters; the first column contains the numerical indices of the parameters; the second and third columns, present for structure based parameter handling, Parameter structures, contain the names of the parameters and the subindices of the parameters, Non-scalar parameters, respectively.  The default gradient function will call the objective function with the second argument set with fields f: as the f passed to the gradient function, plabels: cell-array of 1x1 cell-arrays with the entries of the column-cell-arrays of plabels as passed to the jacobian function corresponding to current parameter, side: 0 for one-sided interval, 1 or 2, respectively, for the sides of a two-sided interval, and parallel: logical scalar indicating parallel computation of partial derivatives.  This information can be useful if the model function can omit some computations depending on the currently computed partial derivative.

objf_hessian
     Function computing the Hessian of the objective function with respect to the parameters.  The default is backend specific.  Will be called with the column vector of parameters as argument.

inverse_hessian
     Logical scalar, indicating whether the Hessian function passed by the user actually returns the inverse of the Hessian.

complex_step_derivative_objf
     Logical scalar, default: false.  Estimate gradient of objective function with complex step derivative approximation.  Use only if you know that your objective function is suitable for this.  No user function for the gradient (objf_grad) must be specified.

save_state
     String with path to a file which will be created for periodical saving of the state of optimization.  Useful for very long optimizations which might get interrupted.  The format of the saved state will be backend-specific.  Currently, only the "siman" backend honours this option.  Default: empty string, meaning no saving of state.

recover_state
     String with path to a file created due to option save_state, which is used to recover a saved state before starting optimization. Default: empty string, meaning no recovering of state.

Structure based parameter handling
----------------------------------

Please Parameter structures.

Backend information
-------------------

Please Scalar optimization and choose backend from menu.

Default backend lm_feasible of scalar optimization
==================================================

A Levenberg/Marquardt-like algorithm, attempting to honour constraints throughout the course of optimization.  This means that the initial parameters must not violate constraints (to find an initial feasible set of parameters, e.g. core Octaves sqp can be used ( octave_sqp), by specifying an objective function which is constant or which returns a norm of the distances to the initial values).  The Hessian is either supplied by the user or is approximated by the BFGS algorithm.  Core Octaves sqp performed better in some tests with unconstrained problems.

Returned value CVG will be 2 or 3 for success and 0 or -4 for failure ( nonlin_min for meaning). Returned structure OUTP will have the fields niter, nobjf, and user_interaction.

Backend-specific defaults are: MaxIter: 20, fract_prec: zeros (size (parameters)), max_fract_change: Inf for all parameters. The setting TolX is not honoured.

Interpretation of Display: if set to "iter", currently only information on applying max_fract_change is printed.

Backend wrapping core Octaves sqp function
==========================================

This backend calls the sqp function of core Octave, a sequential quadratic programming algorithm with BFGS, so that it is usable with the nonlin_min frontend of the optim package. sqp honours constraints for the returned result, but not necessarily during the course of optimization.

Compared to calling sqp directly ( sqp (octave)),

  * a different default (numerical) gradient function is used (that of the frontend),

  * bounds are set to +-Inf by default, not to +-realmax.

The value of the additional setting octave_sqp_tolerance, a tolerance to several termination criteria, is passed as the respective argument to Octaves sqp, which provides the default sqrt (eps).  The settings TolFun, TolX, fract_prec, max_fract_change, Display, and user_interaction are not honoured.

Returned value CVG will be 1 for success and 0 or -4 for failure ( nonlin_min for meaning).  Returned structure OUTP will have the fields niter, nobjf, and lambda.

The default of MaxIter is that of Octaves sqp (100).

Simulated annealing backend siman of scalar optimization
========================================================

A simulated annealing (stochastic) optimizer, changing all parameters at once in a single step, so being suitable for non-bound constraints.

No gradient or hessian of the objective function is used. The settings MaxIter, fract_prec, TolFun, TolX, and max_fract_change are not honoured.

Accepts the additional settings T_init (initial temperature, default 0.01), T_min (final temperature, default 1.0e-5), mu_T (factor of temperature decrease, default 1.005), iters_fixed_T (iterations within one temperature step, default 10), max_rand_step (column vector or structure-based configuration of maximum random steps for each parameter, default 0.005 * PIN), stoch_regain_constr (if true, regain constraints after a random step, otherwise take new random value until constraints are met, default false), trace_steps (set field trace of OUTP with a matrix with a row for each step, first column iteration number, second column repeat number within iteration, third column value of objective function, rest columns parameter values, default false), and siman_log (set field log of OUTP with a matrix with a row for each iteration, first column temperature, second column value of objective function, rest columns numbers of tries with decrease, no decrease but accepted, and no decrease and rejected.

Steps with increase diff of objective function are accepted if rand (1) < exp (- diff / T), where T is the temperature of the current iteration.

If regaining of constraints failed, optimization will be aborted and returned value of CVG will be 0. Otherwise, CVG will be 1. Returned structure OUTP, additionally to the possible fields trace and log described above, will have the fields niter and user_interaction.

Interpretation of Display: if set to "iter", an informational line is printed after each iteration.

If parallel_local is equivalent to true, the objective function is evaluated for several parameter combinations in parallel. If parallel_local is set to an integer > 1, this is the maximal number of parallel processes; if it is <= 1, the maximal number will be the number of available processor cores.  The course of optimization won't be changed by parallelization, provided the random number generator starts with the same state.  To achieve this, some of the parallel results are discarded, causing the speedup to be smaller if the rate of acceptance of results is high.  Also, due to overhead, there won't be any speedup, but even a slowdown, if the objective function is not computationally extensive enough.

Honours options save_state and recover_state, described for the frontend.

Simulated annealing backend samin of scalar optimization
========================================================

A simulated annealing (stochastic) optimizer, changing each parameter separately, so not suitable for non-bound constraints. Requires specification of lower and upper bounds for each parameter and attempts to scan the whole region so defined.

No gradient or hessian of the objective function is used. The settings fract_prec and max_fract_change are not honoured.

Maximal stepwidth is continuously adjusted according to the ratio of accepted steps. Temperature is increased at the start until accepted steps cover the whole parameter range, if this is not the case at the initial temperature.

Accepts the additional settings T_init (initial temperature, default 0.1), mu_T (factor of temperature decrease, default 1.2), iters_fixed_T (subiterations within one temperature step, default 100), iters_adjust_step (number of subiterations after which maximal stepwidth is adjusted, default 5), niter_check_tolfun (number of iterations to compare with if checking the value of the objective function for convergence, default 5), trace_steps (set field trace of OUTP with a matrix with a row for each step, first column iteration number, second column repeat number within iteration, third column value of objective function, fourth column temperature, rest columns parameter values, default false), and siman_log (set field log of OUTP with a matrix with a row for each iteration, first column temperature, second column value of objective function, rest columns numbers of tries with decrease, no decrease but accepted, no decrease and rejected, steps corrected back into bounds, and steps yielding a new best value of the objective function).

Steps with increase diff of objective function are accepted if rand (1) < exp (- diff / T), where T is the temperature of the current iteration.

If converged, the returned value of CVG will be 1, if MaxIters is exceeded before convergence, CVG will be 0. The returned structure OUTP, additionally to the possible fields trace and log described above, will have the fields niter and user_interaction.

Interpretation of Display: if set to "iter", informational text is printed after each iteration and at the end of optimization. If set to "final", informational text is printed only at the end of optimization.

No parallel execution is performed in this backend.

Options save_state and recover_state are not honoured.

Unconstrained Newton-like optimization
======================================

This backend features a Newton-like algorithm. The user has to supply a Hessian function. No constraints are honoured. If the supplied Hessian function actually returns the inverse of the Hessian, set inverse_hessian to true. Supplying the inverse Hessian is preferable, if possible.

Returned value CVG will be 2 or 3 for success and 0 or -1 for failure ( nonlin_min for meaning). Returned structure OUTP will have the fields niter, nobjf, and user_interaction.

Interpretation of Display: if set to "iter", some diagnostics are printed.

A multidirectional search algorithm
===================================

Helptext:
---------

(mdsmax)

An alternating directions algorithm
===================================

Helptext:
---------

(adsmax)

Another Nelder-Mead algorithm
=============================

This function does gradient-less minimization using the Nelder-Mead algorithm. No constraints are honoured.

Helptext:
---------

(nelder_mead_min)

Direction-set (Powell's) method
===============================

(powell)

Unconstrained BFGS algorithm
============================

BFGS or limited memory BFGS minimization of a function. No constraits are honoured.

Helptext:
---------

(bfgsmin)

Newton-Raphson algorithm
========================

No constraints are honoured.

(nrm)

A conjugate gradient method
===========================

(cg_min)

Linesearch, Brent method
========================

(brent_line_min)

Linesearch, minimize a function along dx
========================================

Helptext:
---------

(line_min)

A differential evolution (stochastic) optimizer
===============================================

Helptext:
---------

(de_min)

Repeatedly call bfgsmin
=======================

Helptext:
---------

(battery)

Functions for optimization of a model function returning an array
*****************************************************************

Model functions whose parameters are to be optimized may return a vector or array of values.  Either these or their differences to some constant values (curve fitting) can be minimized in some sense, often, but not necessarily, by minimizing the sum of their squares.  It is usually preferable to use optimizers designed for residual optimization for this purpose.  These can exploit information contained in the individual elements of the returned array, which would not be possible if the user calculated a norm (e.g. sum of squares) of the elements and performed a scalar optimization.

Frontend nonlin_residmin for non-linear residual minimization
=============================================================

(nonlin_residmin)

Settings
--------

The fields of the SETTINGS structure can be set with optimset (octave).

For settings common to all frontends (including these for statistics) Common frontend options.

For additional settings common to all optimization frontends Common optimization options.

Specific defaults:
..................

Algorithm:
"lm_svd_feasible"

Additional settings:
....................

weights
     Array of weights for the residuals. Dimensions must match.

dfdp
     Function computing the Jacobian of the residuals with respect to the parameters, assuming residuals are reshaped to a column vector. Default: real finite differences.  Will be called with the column vector of parameters and, if it accepts it, an informational structure as arguments.  If dfdp was specified by the user, the informational structure has the fields f: value of residuals for current parameters, reshaped to a column vector, fixed: logical vector indicating which parameters are not optimized, so these partial derivatives need not be computed and can be set to zero, diffp, diff_onesided, lbound, ubound: identical to the user settings of this name, plabels: 1-dimensional cell-array of column-cell-arrays, each column with labels for all parameters; the first column contains the numerical indices of the parameters; the second and third columns, present for structure based parameter handling, Parameter structures, contain the names of the parameters and the subindices of the parameters, Non-scalar parameters, respectively.  The default jacobian function will call the model function with the second argument set with fields f: as the f passed to the jacobian function, plabels: cell-array of 1x1 cell-arrays with the entries of the column-cell-arrays of plabels as passed to the jacobian function corresponding to current parameter, side: 0 for one-sided interval, 1 or 2, respectively, for the sides of a two-sided interval, and parallel: logical scalar indicating parallel computation of partial derivatives.  This information can be useful if the model function can omit some computations depending on the currently computed partial derivative.

complex_step_derivative_f
     Logical scalar, default: false. Estimate Jacobian of model function with complex step derivative approximation. Use only if you know that your model function is suitable for this. No user function for the Jacobian (dfdp) must be specified.

plot_cmd
     Function enabling backend to plot results or intermediate results.  Will be called with current computed residuals.  Default: plot nothing.  This setting is deprecated and will disappear.  Please use user_interaction instead ( Common optimization options).

Structure based parameter handling
----------------------------------

Please Parameter structures.

Backend information
-------------------

Please Residual optimization and choose backend from menu under `Optimization backends'.

Function nonlin_curvefit() for curve fitting
============================================

In curve fitting, the model function computes values from a constant set of `independents', and the intention is to minimize the differences of these computed values to a constant set of `observations'. This can be done with nonlin_residmin, but it is more convenient to use nonlin_curvefit, which cares for passing the constant `independents' to the model function and for calculating the differences to the constant `observations'.

However, if in some optimization problem you notice that you end up with passing dummy-values for the `independents' and zeros for the `observations', you can more naturally use nonlin_residmin instead of nonlin_curvefit.

(nonlin_curvefit)

Also, if the setting user_interaction is given, additional information is passed to these functions, Common optimization options.

Default backend lm_svd_feasible of residual minimization
========================================================

Levenberg/Marquardt algorithm using singular value decomposition. Constraints must be met by the initial parameters and are attempted to be kept met throughout the optimization.

Returned value CVG will be 0, 1, or 2. Returned structure OUTP will have the fields niter and user_interaction.

Backend-specific defaults are: MaxIter: 20, fract_prec: zeros (size (parameters)), max_fract_change: Inf for all parameters. The setting TolX is not honoured.

Interpretation of Display: if set to "iter", currently some diagnostics are printed.

Specific option: lm_svd_feasible_alt_s: if falling back to nearly gradient descent, do it more like original Levenberg/Marquardt method, with descent in each gradient component; for testing only.

Statistics for residual minimization
====================================

(residmin_stat)

Further settings
----------------

The fields of the SETTINGS structure can be set with optimset (octave).

For settings common to all frontends Common frontend options.

Additional settings:
....................

objf_type
     Type of objective function of the optimization; must be specified in many cases. This determines which backends to use. Currently, there are only backends for the type "wls" (weighted least squares).

residuals

covd
     Optional information on the result of optimization, residuals and covariance matrix of data, respectively.

weights
     Array of weights applied to the residuals in the previous optimization. Dimensions must match those of the residuals.

dfdp
     Can be set in the same way and has the same default as in nonlin_residmin ( nonlin_residmin), but alternatively may already contain the computed Jacobian of the model function at the final parameters in matrix- or structure-form.

complex_step_derivative_f
     Estimate Jacobian of model function with complex step derivative approximation. Use only if you know that your model function is suitable for this. No user function for the Jacobian (dfdp) must be specified.

Structure based parameter handling
----------------------------------

Please Parameter structures.

Backend information
-------------------

Please Residual optimization and choose backend from menu under `Statistics backends'.

Statistics for curve fitting
============================

As nonlin_curvefit can be used instead of nonlin_residmin for curve fitting ( nonlin_curvefit, nonlin_residmin), curvefit_stat can be used instead of residmin_stat ( residmin_stat) for statistics on the results of curve fitting.

(curvefit_stat)

Statistics for weighted least squares
=====================================

The backends for objf_type == "wls" (currently the only supported type of objective function) compute covd (due to user request or as a prerequisite for covp and corp) as a diagonal matrix by assuming that the variances of data points are proportional to the reciprocal of the squared weights and guessing the factor of proportionality from the residuals. If covp is not defined (e.g. because the Jacobian has no full rank), an attempt is made to still compute its uniquely defined elements, if any. In corp, interdependent parameters can cause elements of 1 or -1, which in this case are not the real coefficients of correlation, but rather indicate the direction of parameter interdependence. To be consistent with this, an attempt is made (often not successful) to identify parameter interdependence and mark it with elements of 1 or -1 in corp even if the respective elements of covp can not be computed.

Linear least squares with linear constraints.
=============================================

(This function does not fit well into this chapter because it is actually a special case of quadratic programming).

(lsqlin)

An older function for curve fitting
===================================

This was a popular function for curve fitting and has been enhanced to honour constraints.  nonlin_curvefit ( nonlin_curvefit) does now the same job if used with the default backend, and should be prefered due to its more powerful interface.  The statistics returned by leasqr can also (and partially better) be computed with curvefit_stat ( curvefit_stat).  There are currently two things which still only leasqr does:

  * internally providing a function for plotting fits during optimization,

  * returning a pre-computed matrix for determining confidence regions.

(leasqr)

Prony's method for non-linear exponential fitting
=================================================

Helptext:
---------

(pronyfit)

Function polyfitinf for polynomial fitting.
===========================================

Helptext:
---------

(polyfitinf)

Polynomial fitting suitable for polyconf
========================================

(wpolyfit)

See also polyfit (octave).

Confidence and prediction intervals for polynomial fitting
==========================================================

Helptext:
---------

(polyconf)

Function LinearRegression
=========================

(LinearRegression)

See also ols (octave), gls (octave), regress (octave), leasqr, nonlin_curvefit, polyfit (octave), wpolyfit, pronyfit.

Function wsolve, another linear solver
======================================

Helptext:
---------

(wsolve)

Functions for finding the zero of a nonlinear user function
***********************************************************

There is only one dedicated zero finder in the optim package, which is just a vectorized version of Octaves fzero ( fzero (octave)).

A vectorized version of fzero
=============================

(vfzero)

Functions for numerical approximation of gradients and Hessians
***************************************************************

You should not usually need to use these functions directly or pass them as arguments. They should be chosen and used by optimizer functions, possibly subject to their configuration options.

Direct user interface to default numerical gradient method of new frontends
===========================================================================

Helptext:
---------

(dfpdp)

Higher order numerical derivatives
==================================

(deriv)

Another numerical gradient function
===================================

Helptext:
---------

(numgradient)

Numerical Hessian function
==========================

Helptext:
---------

(numhessian)

A string, yielding the numerical gradient if evaluated
======================================================

Helptext:
---------

(cdiff)

Complex step derivatives
========================

(jacobs)

Functions for algebraic tasks common to optimization problems
*************************************************************

A complementary pivoting algorithm
==================================

Helptext:
---------

(cpiv_bard)

Gauss-Jordan pivoting
=====================

Helptext:
---------

(gjp)

Function optim_doc to view documentation
****************************************

(optim_doc)

Traditional functions
*********************

Linear programming
==================

This function works by calling glpk of core Octave.

(linprog)

See also glpk (octave).

Quadratic programming
=====================

This function is similar to qp of core Octave.

(quadprog)

Non-linear residual minimization
================================

This function is for Matlab compatibility. It attempts to work like lsqnonlin by calling nonlin_residmin.

(lsqnonlin)

Curve fitting
=============

This function is for Matlab compatibility. It attempts to work like lsqcurvefit by calling nonlin_curvefit.

(lsqcurvefit)

Non-linear regression
=====================

This function is for Matlab compatibility. It attempts to work like nlinfit by calling nonlin_curvefit and curvefit_stat.

(nlinfit)

Non-linear scalar optimization
==============================

(fmincon)

Options common to all frontends
*******************************

All frontends for optimization and for result statistics (nonlin_min, nonlin_residmin, nonlin_curvefit, residmin_stat, curvefit_stat)accept the following options, settable with optimset (octave).

These options are handled within the frontend.

FinDiffRelStep
     Column vector (or scalar, for all parameters) of fractional intervals supposed to be used by gradient or Jacobian functions performing finite differencing. Default: .002 * ones (size (parameters)) for central intervals and .001 * ones (size (parameters)) for one-sided intervals. The default function for finite differencing won't let the absolute interval width get smaller than abs (FinDiffRelStep .* TypicalX (see below).

diffp
     Can be used alternatively to FinDiffRelStep, but for central intervals twice the specified value will be used for backwards compatibility.

diff_onesided
     Logical column vector (or scalar, for all parameters) indicating the parameters for which one-sided intervals (instead of central intervals) should be used by gradient or Jacobian functions performing finite differencing. Default: false (size (parameters)).

FinDiffType
     Can be used alternatively to diff_onesided, but always applies to all parameters at once. Possible values: "central" (central intervals) or "forward" (one-sided intervals).

TypicalX
     Column vector (or scalar, for all parameters) whose absolute value specifies minimal absolute parameter values for computation of intervals in finite differencing by gradient or Jacobian functions (see FinDiffRelStep). Default: 0.0001. Must not be zero.

cstep
     Scalar step size for complex step derivative approximation of gradients or Jacobians. Default: 1e-20.

parallel_local
     Logical or numeric scalar, default: false. If the parallel package, version >= 2.0.5, is loaded, estimate gradients of objective function and Jacobians of model function and of constraints in parallel processes. If parallel_local is set to an integer > 1, this is number of parallel processes; if it is <= 1, the number of processes will be the number of available processor cores. Works for default (real) finite differences and for complex step derivatives. Due to overhead, a speed advantage can only be expected if objective function, model function or constraint functions are time consuming enough. Additionally, this setting is also passed to the individual optimization backends, which may also consider this option (see documentation of backends). If this option is equivalent to true, a warning (ID: optim:parallel_local) will be issued if no parallel package of a correct version is loaded.

parallel_net
     Empty (default) or a parallel connections object, see function pconnect of the parallel package. If not empty, estimate gradients of objective function and Jacobians of model function and of constraints using parallel processing in a network of machines. The considerations regarding a speed advantage are similar to those for option parallel_local.

fixed
     Logical column vector indicating which parameters are not optimized, but kept to their inital value.

Options common to all optimization frontends
********************************************

All frontends for optimization (nonlin_min, nonlin_residmin, nonlin_curvefit) accept the following options, settable with optimset (octave).

Settings handled within the frontend
------------------------------------

Algorithm
     String specifying the backend.

complex_step_derivative_inequc,

complex_step_derivative_equc
     Logical scalars, default: false. Estimate Jacobian of general inequality constraints and equality constraints, respectively, with complex step derivative approximation. Use only if you know that your function of general inequality constraints or function of general equality constraints, respectively, is suitable for this. No user function for the respective Jacobian must be specified.

Settings passed to the backend
------------------------------

Which of these options are actually honored is noted in the descriptions of the individual backends.

lbound,

ubound
     Column vectors of lower and upper bounds for parameters.  Default: -Inf and +Inf, respectively.  The bounds are non-strict, i.e. parameters are allowed to be exactly equal to a bound.  The default function for gradients or Jacobians will respect bounds (but no further inequality constraints) in finite differencing if the backend respects bounds even during the course of optimization.

inequc
     Further inequality constraints.  Cell-array containing up to four entries, two entries for linear inequality constraints and/or one or two entries for general inequality constraints.  Either linear or general constraints may be the first entries, but the two entries for linear constraints must be adjacent and, if two entries are given for general constraints, they also must be adjacent.  The two entries for linear constraints are a matrix (say m) and a vector (say v), specifying linear inequality constraints of the form m.' * parameters + v >= 0.  The first entry for general constraints must be a differentiable column-vector valued function (say h), specifying general inequality constraints of the form h (p[, idx]) >= 0; p is the column vector of optimized paraters and the optional argument idx, given only if the function accepts it, is a logical index.  h has to return the values of all constraints if idx is not given.  It may choose to return only the indexed constraints if idx is given (so computation of the other constraints can be spared); in this case, the additional setting f_inequc_idx has to be set to true.  In gradient determination, this function may be called with an informational third argument (only if the function accepts it), whose content depends on the function for gradient determination.  If a second entry for general inequality constraints is given, it must be a function computing the jacobian of the constraints with respect to the parameters.  For this function, the description of the setting dfdp, dfdp, applies, with 2 exceptions: 1) it is called with 3 arguments (but with the 2nd and 3rd argument only if it accepts them) since it has an additional argument idx, a logical index, at second position, indicating which rows of the jacobian must be returned (if the function chooses to return only indexed rows, the additional setting df_inequc_idx has to be set to true).  2) the default jacobian function calls h with 3 arguments, since the argument idx is also supplied.  Note that specifying linear constraints as general constraints will generally waste performance, even if further, non-linear, general constraints are also specified.

f_inequc_idx,

df_inequc_idx
     Indicate that functions for general inequality constraints or their jacobian, respectively, return only the values or derivatives for the indexed parameters.  See description of setting inequc above.

equc
     Equality constraints.  Specified the same way as inequality constraints (see inequc above).

f_equc_idx,

df_equc_idx
     As f_inequc_idx and df_inequc_idx above, but for equality constraints.

cpiv
     Function for complementary pivoting, usable in algorithms for constraints.  Default: cpiv_bard.  Only the default function is supplied with the package.

TolFun
     Minimum fractional improvement in objective function (e.g. sum of squares) in an iteration (termination criterium).  Default: .0001.

TolX
     Minimum fractional change in a norm of the parameters in an iteration (termination criterium).  Default: backend specific.

MaxIter
     Maximum number of iterations (termination criterium).  Default: backend-specific.

fract_prec
     Column Vector, minimum fractional changes of corresponding parameters in an iteration (termination criterium if violated in two consecutive iterations).  Default: backend-specific.

max_fract_change
     Column Vector, enforced maximum fractional changes in corresponding parameters in an iteration. Default: backend-specific.

Display
     String indicating the degree of verbosity. Default: "off". Possible values are currently "off" (no messages) and "iter" (some messages after each iteration).  Support of this setting and its exact interpretation are backend-specific.

debug
     Logical scalar, default: false. Will be passed to the backend, which might print debugging information if true.

FunValCheck
     If "on", the output of user functions will be sanity-checked. Default: "off".

user_interaction
     Handle to a user function or cell-array with a number of these. Functions must have this interface:

          [STOP, INFO] = some_user_function (P, VALS,
                                                         STATE);

     If STOP is true, the algorithm stops.  In INFO information about the reason for stopping can be returned in a free format.  INFO can be set to be empty, but it must be set.  Note that this is different from the otherwise similar Matlab setting OutputFcn.  The functions will be called by the algorithms at the start with STATE set to init, after each iteration with STATE set to iter, and at the end with STATE set to done.  P contains the current parameters, and VALS is a structure with other current values, the possible fields are currently:

     iteration
          number of the current iteration,

     fval
          value of objective function (for scalar optimization),

     residual
          residuals (for residual-based optimization),

     model_y
          in nonlin_curvefit, the output of the model function,

     observations
          in nonlin_curvefit, the constant observations,

     model_x
          in nonlin_curvefit, the constant argument X.

     Information about the output of these functions when they were called the last time (possibly causing a stop) will be contained in the output OUTP of the frontend in field user_interaction.  Subfield stop is a vector containing the STOP outputs of each function, subfield info is a cell-array containing the output INFO of each function.  In the case of a stop, the output CVG of the frontent will be -1.

Handling of structures of optimized parameters
**********************************************

It can be convenient not to handle the optimized parameters as elements of a vector, but as named fields of a structure.  The frontends nonlin_residmin, nonlin_curvefit, residmin_stat, curvefit_stat, and nonlin_min can accept parameter information in structure form, and can pass the parameters as a structure to user functions, although the backends still handle the parameters as vectors.

To use this feature, the initial parameters must be given in structure form, or the setting param_order must be given, a cell-array with names of the parameters.  If both is done, only the parameters in structure fields named in param_order will be optimized.  If there are still some non-structure-based configuration settings or user functions, specifying param_order is mandatory even if the initial parameters are given in structure form.

If the initial parameters are a structure, the parameters being the optimization result will also be returned as a structure.

Specify which user functions accept parameter structures
========================================================

The frontend must be told which user functions accept parameter structures by setting corresponding settings to true. The names of these settings depend on which user functions are applicable to the respective frontend and are shown in the following table.

User function   Setting   Frontends
Objective function
objf_pstruct
nonlin_min
Gradient
grad_objf_pstruct
nonlin_min
Hessian
hessian_objf_pstruct
nonlin_min
Model function
f_pstruct
nonlin_residmin, nonlin_curvefit, residmin_stat, curvefit_stat
Jacobian
df_pstruct
nonlin_residmin, nonlin_curvefit, residmin_stat, curvefit_stat
General inequality constraints
f_inequc_pstruct
nonlin_min, nonlin_residmin, nonlin_curvefit
Jacobian of general inequality constraints
df_inequc_pstruct
nonlin_min, nonlin_residmin, nonlin_curvefit
General equality constraints
f_equc_pstruct
nonlin_min, nonlin_residmin, nonlin_curvefit
Jacobian of general equality constraints
df_equc_pstruct
nonlin_min, nonlin_residmin, nonlin_curvefit

Format of returned values of structure-based gradient and Hessian functions
===========================================================================

Structure-based gradient or Jacobian functions, including Jacobians of general constraints, must return the partial derivatives as fields of a structure under the respective parameter names.  For gradients, the partial derivatives are scalar.  For Jacobians, the partial derivatives must be column vectors.

Structure-based Hessian functions must return the 2nd derivatives as subfields in a two-level structure of parameter names.  For example, if the parameter names are a and b, the 2nd derivative with respect to a and b must be in the field returned_structure.a.b or returned_structure.b.a (there is no need to specify both).

Specify structure-based linear constraints
==========================================

Linear constraints, otherwise specified with a matrix and a vector , can be adapted to structure-based parameter handling by specifying, instead of a matrix, a structure containing the rows of the matrix in fields under the respective parameter names.  In this case, rows containing only zeros need not be given.

Parameter-related configuration settings in structure form
==========================================================

The vector-based settings lbound, ubound, fixed, diffp, diff_onesided, fract_prec, and max_fract_change can be replaced by the setting param_config.  It is a structure that can contain fields corresponding to parameter names.  For each such field, there may be subfields with the same names as the above vector-based settings, but containing a scalar value for the respective parameter.

For example, if the parameters are named a and b, instead of specifying

     settings = optimset ("lbound", [-Inf; 0],
                          "diff_onesided", [true; true]);

one can specify

     pconf.b.lbound = 0;
     pconf.a.diff_onesided = true;
     pconf.b.diff_onesided = true;
     settings = optimset ("param_config", pconf);

If param_config is specified, none of the above vector-based settings may be used.

Handling named parameter arrays
===============================

Parameters in named structure fields are allowed to be non-scalar real arrays.  In this case, their dimensions must be given by the setting param_dims, a cell-array of dimension vectors, each containing at least two dimensions; if not given, dimensions are taken from the initial parameters, if these are given in a structure.

If there are any vector-based settings or not structure-based linear constraints, they must correspond to an order of parameters defined as follows:

All named parameter arrays are reshaped to vectors. Then, all parameters, scalars and vectors, are concatenated in the order of parameter names, given by the user.

Structure-based settings or structure-based initial parameters must contain arrays with dimensions reshapable to those of the respective parameters.

Passing additional parameters to user functions
***********************************************

Optimizers often require the user to supply functions (e.g. objective functions, model functions, constraint functions).  The interface of these functions --- arguments and returned values --- is defined by the optimizer.  Often, a user function needs additional arguments, not covered by the defined interface, which are constant throughout the optimization.  These can be supplied by wrapping the user function into an anonymous function.  See Anonymous Functions (octave), for further explanation and examples.

There are also some older optimizers in the optim package, written when anonymous functions were not available in Octave.  Some of these offer an interface to user functions which is itself able to pass additional constant variables in arbitrary argument positions.  Newer optimizers should not be written this way, since this is an unnecessary complication.

Though it is possible to use global variables to pass additional data to user functions, this is not recommended since it introduces the possibility of name conflicts within the pool of global variables.