#+TITLE: mrcal lens models
#+OPTIONS: toc:t
mrcal supports a wide range of projection models. Some are intended to represent
physical lenses, while others are idealized, useful for processing. /All/ are
referred to as /lens/ models in the code and in the documentation. The
representation details and projection behaviors are described here.

* Representation
:PROPERTIES:
:CUSTOM_ID: representation
:END:

A =mrcal= /lens/ model represents a lens independent of its pose in space. A
lens model is fully specified by

- A model /family/ (or /type/). This is something like =LENSMODEL_PINHOLE= or
  =LENSMODEL_SPLINED_STEREOGRAPHIC=
- /Configuration/ parameters. This is a set of key/value pairs, which is
  required only by some model families. These values are /not/ subject to
  optimization, and may affect how many optimization parameters are needed.
- Optimization parameters. These are the parameters that the optimization
  routine controls as it runs

Each model family also has some /metadata/ key/value pairs associated with it.
These are inherent properties of a model family, and are not settable. At the
time of this writing there are 3 metadata keys:

- =has_core=: True if the first 4 optimization values are the "core": $f_x$,
  $f_y$, $c_x$, $c_y$. As of mrcal 2.3 this is True for all models.
- =can_project_behind_camera=: True if this model is able to project vectors
  from behind the camera. If it cannot, then [[file:mrcal-python-api-reference.html#-unproject][=mrcal.unproject()=]] will never
  report =z= < 0. As of mrcal 2.3 this is True only for models based on a
  stereographic projection, for instance [[#splined-stereographic-lens-model][=LENSMODEL_SPLINED_STEREOGRAPHIC=]]. It
  is False for models based on a pinhole projection, for instance
  =LENSMODEL_OPENCV...= and =LENSMODEL_CAHVOR...=.
- =has_gradients=: True if this model has gradients implemented. As of mrcal 2.3
  all models have this.
- =noncentral=: True if [[file:formulation.org::#lens-behavior][projection is /not/ invariant to scaling; i.e. if every
  observation ray does /not/ intersect at the same point]]. As of mrcal 2.3 only
  [[#lensmodel-cahvore][=LENSMODEL_CAHVORE=]] does this.

In Python, the models are identified with a string =LENSMODEL_XXX= where the
=XXX= selects the specific model family and the configuration, if needed. A
sample model string with a configuration:
=LENSMODEL_SPLINED_STEREOGRAPHIC_order=3_Nx=30_Ny=20_fov_x_deg=170=. The
configuration is the pairs =order=3=, =Nx=30= and so on. At this time, model
families that accept a configuration /require/ it to be specified fully,
although optional configuration keys will be added soon. Today, calling Python
functions with =LENSMODEL_SPLINED_STEREOGRAPHIC= or
=LENSMODEL_SPLINED_STEREOGRAPHIC_order=3= will fail due to an incomplete
configuration. The [[file:mrcal-python-api-reference.html#-lensmodel_metadata_and_config][=mrcal.lensmodel_metadata_and_config()=]] function returns a
dict containing the metadata and configuration for a particular model string.

In C, the model family is selected with the [[https://www.github.com/dkogan/mrcal/blob/master/mrcal.h#mrcal_lensmodel_type_t][=mrcal_lensmodel_type_t=]] enum. The
elements are the same as the Python model names, but with =MRCAL_= prepended. So
the sample model from above has type =MRCAL_LENSMODEL_SPLINED_STEREOGRAPHIC=. In
C the [[https://www.github.com/dkogan/mrcal/blob/master/mrcal.h##mrcal_lensmodel_t][=mrcal_lensmodel_t=]] structure contains the type /and/ configuration. This
structure is thus an analogue the the model strings, as Python sees them. So a
number of C functions accepting [[https://www.github.com/dkogan/mrcal/blob/master/mrcal.h##mrcal_lensmodel_t][=mrcal_lensmodel_t=]] arguments are analogous to
Python functions taking model strings. For instance, the number of parameters
needed to fully describe a given model can be obtained by calling
[[file:mrcal-python-api-reference.html#-lensmodel_num_params][=mrcal.lensmodel_num_params()=]] in Python or [[https://www.github.com/dkogan/mrcal/blob/master/mrcal.h#mrcal_lensmodel_num_params][=mrcal_lensmodel_num_params()=]] in C.
Given a [[https://www.github.com/dkogan/mrcal/blob/master/mrcal.h##mrcal_lensmodel_t][=mrcal_lensmodel_t lensmodel=]] structure of type =XXX= (i.e. if
=lensmodel.type= is =MRCAL_LENSMODEL_XXX=) then the configuration is available
in =lensmodel.LENSMODEL_XXX__config=, which has type
=mrcal_LENSMODEL_XXX__config_t=. The metadata is requestable by calling this
function:

#+begin_src c
mrcal_lensmodel_metadata_t mrcal_lensmodel_metadata( const mrcal_lensmodel_t* lensmodel );
#+end_src

* Intrinsics core
:PROPERTIES:
:CUSTOM_ID: core
:END:
Most models contain an "intrinsics core". These are 4 values that appear at the
start of the parameter vector:

- $f_x$: the focal length in the horizontal direction, in pixels
- $f_y$: the focal length in the vertical direction, in pixels
- $c_x$: the horizontal projection center, in pixels
- $c_y$: the vertical projection center, in pixels

At this time all models contain a core.

* Models
Currently all models except [[#lensmodel-cahvore][=LENSMODEL_CAHVORE=]] represent a /central/
projection: all observation rays intersect at a single point (the camera
origin). So $k \vec v$ projects to the same $\vec q$ for all $k$. This isn't
strictly true for real-world lenses, so [[file:formulation.org::#lens-behavior][non-central projections]] will be
supported in a future release of mrcal.

** =LENSMODEL_PINHOLE=
:PROPERTIES:
:CUSTOM_ID: lensmodel-pinhole
:END:
This is the basic "pinhole" model with 4 parameters: the core. Projection of a
point $\vec p$ is defined as

\[\vec q = \left[ \begin{aligned} f_x \frac{p_x}{p_z} + c_x \\ f_y \frac{p_y}{p_z} + c_y \end{aligned} \right] \]

This model is defined only in front of the camera, and projects to infinity as
we approach 90 degrees off the optical axis ($p_z \rightarrow 0$). Straight
lines in space remain straight under this projection, and observations of the
same plane by two pinhole cameras define a [[https://en.wikipedia.org/wiki/Homography][homography]]. This model can be used
for [[file:stereo.org][stereo rectification]], although it only works well with long lenses. Longer
lenses tend to have roughly pinhole behavior, but no real-world lens follows
this projection, so this exists for data processing only.

** =LENSMODEL_STEREOGRAPHIC=
:PROPERTIES:
:CUSTOM_ID: lensmodel-stereographic
:END:
This is another trivial model that exists for data processing, and not to
represent real lenses. Like the pinhole model, this has just the 4 core
parameters.

To define the projection of a point $\vec p$, let's define the angle off the
optical axis:

\[ \theta \equiv \tan^{-1} \frac{\left| \vec p_{xy} \right|}{p_z} \]

then

\[ \vec u \equiv \frac{\vec p_{xy}}{\left| \vec p_{xy} \right|} 2 \tan\frac{\theta}{2} \]

and

\[\vec q = \left[ \begin{aligned} f_x u_x + c_x \\ f_y u_y + c_y \end{aligned} \right] \]

This model is able to project behind the camera, and has a single singularity:
directly opposite the optical axis. mrcal refers to $\vec u$ as the
/normalized/ stereographic projection; we get the projection $\vec q = \vec u$
when $f_x = f_y = 1$ and $c_x = c_y = 0$

Note that the pinhole model can be defined in the same way, except the pinhole
model has $\vec u \equiv \frac{\vec p_{xy}} {\left| \vec p_{xy} \right|} \tan
\theta$. And we can thus see that for long lenses the pinhole model and the
stereographic model function similarly: $\tan \theta \approx 2 \tan
\frac{\theta}{2}$ as $\theta \rightarrow 0$

** =LENSMODEL_LONLAT=
:PROPERTIES:
:CUSTOM_ID: lensmodel-lonlat
:END:

This is a standard [[https://en.wikipedia.org/wiki/Equirectangular_projection][equirectangular projection]]. It's a trivial model useful
not for representing lenses, but for describing the projection function of wide
panoramic images. This works just like latitude an longitude on a globe, with a
linear angular map on latitude and longitude. The 4 intrinsics core parameters
are used to linearly map latitude, longitude to pixel coordinates. The full
projection expression to map a camera-coordinate point $\vec p$ to an image
pixel $\vec q$:

\[
\vec q =
\left[ \begin{aligned}
f_x \, \mathrm{lon} + c_x \\
f_y \, \mathrm{lat} + c_y
\end{aligned} \right]
=
\left[ \begin{aligned}
f_x \tan^{-1}\left(\frac{p_x}{p_z}\right) + c_x \\
f_y \sin^{-1}\left(\frac{p_y}{\left|\vec p\right|}\right) + c_y
\end{aligned} \right]
\]

So $f_x$ and $f_y$ specify the angular resolution, in pixels/radian.

For normal lens models the optical axis is at $\vec p = \left[ \begin{aligned} 0
\\ 0 \\ 1 \end{aligned} \right]$, and projects to roughly the center of the
image, roughly at $\vec q = \left[ \begin{aligned} c_x \\ c_y \end{aligned}
\right]$. /This/ model has $\mathrm{lon} = \mathrm{lat} = 0$ at the optical
axis, which produces the same, usual $\vec q$. However, this projection doesn't
represent a lens and there is no "camera" or an "optical axis". The view may be
centered anywhere, so $c_x$ and $c_y$ could be anything, even negative.

The special case of $f_x = f_y = 1$ and $c_x = c_y = 0$ (the default values in
[[file:mrcal-python-api-reference.html#-project_lonlat][=mrcal.project_lonlat()=]]) produces a /normalized/ equirectangular projection:

\[
\vec q_\mathrm{normalized} =
\left[ \begin{aligned}
\mathrm{lon} \\\mathrm{lat}
\end{aligned} \right]
\]

This projection has a singularity at the poles, approached as $x \rightarrow 0$
and $z \rightarrow 0$.

** =LENSMODEL_LATLON=
:PROPERTIES:
:CUSTOM_ID: lensmodel-latlon
:END:

This is a "transverse equirectangular projection". It works just like
[[#lensmodel-lonlat][=LENSMODEL_LONLAT=]], but rotated 90 degrees. So instead of a globe oriented as
usual with a vertical North-South axis, this projection has a horizontal
North-South axis. The projected $x$ coordinate corresponds to the latitude, and
the projected $y$ coordinate corresponds to the longitude.

As with [[#lensmodel-lonlat][=LENSMODEL_LONLAT=]], lenses do not follow this model. It is useful as the
core of a [[file:stereo.org][rectified view used in stereo processing]]. The full projection
expression to map a camera-coordinate point $\vec p$ to an image pixel $\vec q$:

\[
\vec q =
\left[ \begin{aligned}
f_x \, \mathrm{lat} + c_x \\
f_y \, \mathrm{lon} + c_y
\end{aligned} \right]
=
\left[ \begin{aligned}
f_x \sin^{-1}\left(\frac{p_x}{\left|\vec p\right|}\right) + c_x \\
f_y \tan^{-1}\left(\frac{p_y}{p_z}\right) + c_y
\end{aligned} \right]
\]

As with [[#lensmodel-lonlat][=LENSMODEL_LONLAT=]], $f_x$ and $f_y$ specify the angular resolution, in
pixels/radian. And $c_x$ and $c_y$ specify the projection at the optical axis
$\vec p = \left[ \begin{aligned} 0 \\ 0 \\ 1 \end{aligned} \right]$.

The special case of $f_x = f_y = 1$ and $c_x = c_y = 0$ (the default values in
[[file:mrcal-python-api-reference.html#-project_latlon][=mrcal.project_latlon()=]]) produces a /normalized/ transverse equirectangular
projection:

\[
\vec q_\mathrm{normalized} =
\left[ \begin{aligned}
\mathrm{lat} \\\mathrm{lon}
\end{aligned} \right]
\]

This projection has a singularity at the poles, approached as $y \rightarrow 0$
and $z \rightarrow 0$.

** =LENSMODEL_OPENCV4=, =LENSMODEL_OPENCV5=, =LENSMODEL_OPENCV8=, =LENSMODEL_OPENCV12=
:PROPERTIES:
:CUSTOM_ID: lensmodel-opencv
:END:
These are simple parametric models that have the given number of "distortion"
parameters in addition to the 4 core parameters. The projection behavior is
described in the [[https://docs.opencv.org/4.5.0/d9/d0c/group__calib3d.html#details][OpenCV documentation]]. These do a reasonable job in representing
real-world lenses, /and/ they're compatible with many other tools. The
projection function is

\begin{align*}
\vec P &\equiv \frac{\vec p_{xy}}{p_z} \\
r &\equiv \left|\vec P\right|            \\
\vec P_\mathrm{radial} &\equiv \frac{ 1 + k_0 r^2 + k_1 r^4 + k_4 r^6}{ 1 + k_5 r^2 + k_6 r^4 + k_7 r^6} \vec P \\
\vec P_\mathrm{tangential} &\equiv
  \left[ \begin{aligned}
    2 k_2 P_0 P_1 &+ k_3 \left(r^2 + 2 P_0^2 \right) \\
    2 k_3 P_0 P_1 &+ k_2 \left(r^2 + 2 P_1^2 \right)
  \end{aligned}\right] \\
\vec P_\mathrm{thinprism} &\equiv
  \left[ \begin{aligned}
    k_8    r^2 + k_9    r^4 \\ 
    k_{10} r^2 + k_{11} r^4
  \end{aligned}\right] \\
\vec q &= \vec f_{xy} \left( \vec P_\mathrm{radial} + \vec P_\mathrm{tangential} + \vec P_\mathrm{thinprism} \right) + \vec c_{xy}
\end{align*}

The parameters are $k_i$. For any N-parameter OpenCV model the higher-order
terms $k_i$ for $i \geq N$ are all 0. So the tangential distortion terms exist for
all the models, but the thin-prism terms exist only for =LENSMODEL_OPENCV12=.
The radial distortion is a polynomial in =LENSMODEL_OPENCV4= and
=LENSMODEL_OPENCV5=, but a rational for the higher-order models.
Practically-speaking =LENSMODEL_OPENCV8= works decently well for wide lenses.
For non-fisheye lenses, =LENSMODEL_OPENCV4= and =LENSMODEL_OPENCV5= work ok. I'm
sure scenarios where =LENSMODEL_OPENCV12= is beneficial exist, but I haven't
come across them.

** =LENSMODEL_CAHVOR=
:PROPERTIES:
:CUSTOM_ID: cahvor-lens-model
:END:
mrcal supports =LENSMODEL_CAHVOR=, a lens model used in a number of tools at
JPL. The =LENSMODEL_CAHVOR= model has 5 "distortion" parameters in addition to
the 4 core parameters. This support exists only for compatibility, and there's
no reason to use it otherwise. This model is described in:

[[https://trs.jpl.nasa.gov/bitstream/handle/2014/20686/98-1739.pdf][Gennery, D.B. 2001. Least-squares camera calibration including lens distortion
and automatic editing of calibration points. In Calibration and Orientation of
Cameras in Computer Vision, A. Gruen and T.S. Huang (Eds.), Springer-Verlag, pp.
123–136]]

** =LENSMODEL_CAHVORE_...=
:PROPERTIES:
:CUSTOM_ID: lensmodel-cahvore
:END:
This is an extended flavor of =LENSMODEL_CAHVOR= to support wider lenses. The
=LENSMODEL_CAHVORE= model has 8 "distortion" parameters in addition to the 4
core parameters. As with =LENSMODEL_CAHVOR= this support exists only for
compatibility, and there's no reason to use it otherwise.

CAHVORE is a [[file:formulation.org::#lens-behavior][noncentral projection]], so more infrastructure is required to make
=unproject()= do something reasonable in all cases. Thus projection uncertainty
and differencing don't work for this model, and throw an error.

*** Configuration
=LENSMODEL_CAHVORE= models have a single configuration parameter: =linearity=.

- =linearity= = 0 is a "fisheye" CAHVORE model. This is referred to as
  =CAHVORE2= in JPL tools
- =linearity= = 1 is a "perspective" CAHVORE model. This is referred to as
  =CAHVORE1= in JPL tools
- Any other =linearity= is a "general" CAHVORE model. This is referred to as
  =CAHVORE3= in JPL tools

*** Some notes

- The last 3 intrinsic parameters of =LENSMODEL_CAHVORE= are the $\vec E$
  vector. If these are 0, this becomes a central projection, and
  [[file:mrcal-python-api-reference.html#-unproject][=mrcal.unproject()=]] is able to work normally. If you want to compute
  differences you need [[file:mrcal-python-api-reference.html#-unproject][=mrcal.unproject()=]], so you must "centralize" the model
  by setting =intrinsics[-3:] = 0=. This ignores any noncentral effects, which
  could be significant close to the lens. Doing this also invalidates the
  uncertainty logic. More general noncentral projection logic will be added in a
  future release of mrcal, and this situation may improve then

- A =LENSMODEL_CAHVORE= projection with $\vec E = 0$ and =linearity= = 1 is
  equivalent to a =LENSMODEL_CAHVOR= projection

This model is described in:

[[https://link.springer.com/article/10.1007/s11263-006-5168-1][Gennery, D.B. Generalized Camera Calibration Including Fish-Eye Lenses. Int J
Comput Vision 68, 239–266 (2006)]]

** =LENSMODEL_SPLINED_STEREOGRAPHIC_...=
:PROPERTIES:
:CUSTOM_ID: splined-stereographic-lens-model
:END:

This is a stereographic model with correction factors. It is mrcal's attempt to
model real-world lens behavior with more fidelity than the usual parametric
models make possible.

To compute a projection using this model, we first compute the normalized
stereographic projection $\vec u$ as in the [[#lensmodel-stereographic][=LENSMODEL_STEREOGRAPHIC=]] definition
above:

\[ \theta \equiv \tan^{-1} \frac{\left| \vec p_{xy} \right|}{p_z} \]

\[ \vec u \equiv \frac{\vec p_{xy}}{\left| \vec p_{xy} \right|} 2 \tan\frac{\theta}{2} \]

Then we use $\vec u$ to look-up a $\Delta \vec u$ using two separate splined
surfaces:

\[ \Delta \vec u \equiv
\left[ \begin{aligned}
\Delta u_x \left( \vec u \right) \\
\Delta u_y \left( \vec u \right)
\end{aligned} \right] \]

and we then define the rest of the projection function:

\[\vec q =
 \left[ \begin{aligned}
 f_x \left( u_x + \Delta u_x \right) + c_x \\
 f_y \left( u_y + \Delta u_y \right) + c_y
\end{aligned} \right] \]

The $\Delta \vec u$ are the off-stereographic terms. If $\Delta \vec u = 0$, we
get a plain stereographic projection.

Much more detail about this model is available on the [[file:splined-models.org][splined models page]].

*** Configuration
There are several configuration parameters described on the [[file:splined-models.org::#splined-models-configuration-selection][splined models page]].