Levenberg Marquardt Optimizer

LMoptimizer

A Levenberg-Marquardt optimizer for nonlinear least squares problems.

This class implements the Levenberg-Marquardt optimization algorithm to solve problems of the form min_x || model(x) - y_data ||^2, where model(x) is a nonlinear function and y_data is the observed data. It combines gradient descent and Gauss-Newton methods to adaptively optimize the model parameters x.

Attributes:

Name	Type	Description
model	Callable	The nonlinear function to optimize, which should return a vector.
bounds	Tensor	The bounds for each parameter of the optimization.
x	Tensor	The current guess for the parameters.
initial_guess	Tensor	The initial guess for the parameters.
best_x	Tensor	The best found parameters during the optimization.
y_data	Tensor	The observed data.
_lambda	float	The damping factor for the Levenberg-Marquardt update.
options		(LMOptions): Optimizer parameters (see LMOptions)
parameters	List[Tensor]	List of best parameters at each optimization iteration
error_record	List[float]	List of error at each optimization iteration
error	List[float]	List of best error at each optimization iteration

Methods:

Name	Description
step	Optional[Callable] = None) -> None: Performs one optimization step, adjusting the parameters based on the current Jacobian and error.

__init__(model, bounds, initial_guess, y_data, options=LMoptions())

Initializes the Levenberg-Marquardt optimizer with given parameters.

Parameters:

Name	Type	Description	Default
model	Callable	The function to be optimized. Takes an input array and returns an output array.	required
bounds	Tensor	Bounds to constrain the input guesses. Each element should be a (lower bound, upper bound) tuple. Use empty tuples for unbounded inputs.	required
initial_guess	Tensor	Initial guess for the input.	required
y_data	Tensor	Observed data for comparison with the function output. Levenberg-Marquardt will attempt to match to this data.	required
options	LMoptionsTorch	Configuration options for the Levenberg-Marquardt optimization algorithm.	LMoptions()

get_error(y, y_data)

Computes the squared error between the predicted and observed data.

Parameters:

Name	Type	Description	Default
y	Tensor	Predicted data.	required
y_data	Tensor	Target output.	required

Returns:

Name	Type	Description
e	float	Squared error between the predicted and observed data.

step()

Executes a single iteration of the Levenberg-Marquardt optimization process.

LMoptions

Configuration options for the Levenberg-Marquardt optimization algorithm.

Attributes:

Name	Type	Description
jac_delta	float	Step size used to compute the Jacobian via finite differences. Default is 1e-4.
jac_update_interval	int	Number of accepted steps between updates of the Jacobian. Default is 1.
bad_run_jac_interval_weight	float	Weight applied to the Jacobian update interval when a bad step occurs. Default is 0.25.
lambda_init	float	Initial value for the lambda parameter, dictating the balance between gradient descent and Gauss-Newton. Default is 1.
lambda_upscale_factor	float	Factor to increase the lambda parameter when a bad step occurs, making behavior more similar to gradient descent. Default is 11.
lambda_downscale_factor	float	Factor to decrease the lambda parameter when a good step occurs, making behavior more similar to Gauss-Newton. Default is 9.
max_initial_delta	float	Maximum initial step size relative to parameter bounds. Default is 0.1.
jac_function	Optional[Callable]	Custom function to compute the Jacobian. If None, the Jacobian is computed numerically. Default is None.
accept_thresh	float	The threshold for accepting a step based on the error reduction.