1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
|
#+title: A tour of mrcal: differencing
#+OPTIONS: toc:nil
* Previous
We just [[file:tour-initial-calibration.org][gathered calibration images and computed some calibrations]].
* Differencing
An overview follows; see the [[file:differencing.org][differencing page]] for details.
We just used the same chessboard observations to compute the intrinsics of a
lens in two different ways:
- Using a lean =LENSMODEL_OPENCV8= lens model
- Using a rich splined-stereographic lens model
And we saw evidence that the splined model does a better job of representing
reality. Can we quantify that?
Let's compute the difference. There's an obvious algorithm: given a pixel $\vec
q_0$ we
1. Unproject $\vec q_0$ to a fixed point $\vec p$ using lens 0
2. Project $\vec p$ back to pixel coords $\vec q_1$ using lens 1
3. Report the reprojection difference $\vec q_1 - \vec q_0$
[[file:figures/diff-notransform.svg]]
This is a very common thing to want to do, so mrcal provides a [[file:mrcal-show-projection-diff.html][tool]] to do it.
Let's compare the two models:
#+begin_src sh
mrcal-show-projection-diff \
--intrinsics-only \
--cbmax 15 \
--unset key \
opencv8.cameramodel \
splined.cameramodel
#+end_src
#+begin_src sh :exports none :eval no-export
mkdir -p ~/projects/mrcal-doc-external/figures/diff
D=~/projects/mrcal-doc-external/2022-11-05--dtla-overpass--samyang--alpha7/2-f22-infinity/
mrcal-show-projection-diff \
--intrinsics-only \
--cbmax 15 \
--unset key \
$D/{opencv8,splined}.cameramodel \
--hardcopy ~/projects/mrcal-doc-external/figures/diff/diff-radius0-heatmap-splined-opencv8.png \
--terminal 'pngcairo size 1024,768 transparent noenhanced crop font ",12"'
mrcal-show-projection-diff \
--intrinsics-only \
--cbmax 15 \
--unset key \
$D/{opencv8,splined}.cameramodel \
--hardcopy ~/projects/mrcal-doc-external/figures/diff/diff-radius0-heatmap-splined-opencv8.pdf \
--terminal 'pdf size 8in,6in noenhanced solid color font ",16"'
pdfcrop ~/projects/mrcal-doc-external/figures/diff/diff-radius0-heatmap-splined-opencv8.pdf
#+end_src
[[file:external/figures/diff/diff-radius0-heatmap-splined-opencv8.png]]
Well that's strange. The reported differences really do have units of /pixels/.
Are the two models really /that/ different? If we ask for the vector field of
differences instead of a heat map, we get a hint about what's going on:
#+begin_src sh
mrcal-show-projection-diff \
--intrinsics-only \
--vectorfield \
--vectorscale 10 \
--gridn 30 20 \
--cbmax 15 \
--unset key \
opencv8.cameramodel \
splined.cameramodel
#+end_src
#+begin_src sh :exports none :eval no-export
D=~/projects/mrcal-doc-external/2022-11-05--dtla-overpass--samyang--alpha7/2-f22-infinity/
mrcal-show-projection-diff \
--intrinsics-only \
--vectorfield \
--vectorscale 10 \
--gridn 30 20 \
--cbmax 15 \
--unset key \
$D/{opencv8,splined}.cameramodel \
--hardcopy ~/projects/mrcal-doc-external/figures/diff/diff-radius0-vectorfield-splined-opencv8.svg \
--terminal 'svg size 800,450 noenhanced solid dynamic font ",14"'
mrcal-show-projection-diff \
--intrinsics-only \
--vectorfield \
--vectorscale 10 \
--gridn 30 20 \
--cbmax 15 \
--unset key \
$D/{opencv8,splined}.cameramodel \
--hardcopy ~/projects/mrcal-doc-external/figures/diff/diff-radius0-vectorfield-splined-opencv8.pdf \
--terminal 'pdf size 8in,6in noenhanced solid color font ",16"'
pdfcrop ~/projects/mrcal-doc-external/figures/diff/diff-radius0-vectorfield-splined-opencv8.pdf
#+end_src
[[file:external/figures/diff/diff-radius0-vectorfield-splined-opencv8.svg]]
This is a /very/ regular pattern. What does it mean?
The issue is that each calibration produces noisy estimates of all the
intrinsics and all the coordinate transformations:
[[file:figures/uncertainty.svg]]
The above plots projected the same $\vec p$ in the camera coordinate system, but
that coordinate system has shifted between the two models we're comparing. So in
the /fixed/ coordinate system attached to the camera housing, we weren't in fact
projecting the same point.
There exists some transformation between the camera coordinate system from the
solution and the coordinate system defined by the physical camera housing. It is
important to note that *this implied transformation is built-in to the
intrinsics*. Even if we're not explicitly optimizing the camera pose, this
implied transformation is still something that exists and moves around in
response to noise. Rich models like the [[file:splined-models.org][splined stereographic models]] are able to
encode a wide range of implied transformations, but even the simplest models
have some transform that must be compensated for.
The above vector field suggests that we need to pitch one of the cameras. We can
automate this by adding a critical missing step to the procedure above between
steps 1 and 2:
- Transform $\vec p$ from the coordinate system of one camera to the coordinate
system of the other camera
[[file:figures/diff-yestransform.svg]]
We don't know anything about the physical coordinate system of either camera, so
we do the best we can: we compute a fit. The "right" transformation will
transform $\vec p$ in such a way that the reported mismatches in $\vec q$ will
be small. Lots of [[file:differencing.org][details]] are glossed-over here. Previously we passed
=--intrinsics-only= to bypass this fit. Let's omit that option to get the the
diff that we expect:
#+begin_src sh
mrcal-show-projection-diff \
--unset key \
opencv8.cameramodel \
splined.cameramodel
#+end_src
#+begin_src sh :exports none :eval no-export
D=~/projects/mrcal-doc-external/2022-11-05--dtla-overpass--samyang--alpha7/2-f22-infinity/
mrcal-show-projection-diff \
--unset key \
$D/{opencv8,splined}.cameramodel \
--title 'Diff looking at 2 models, computing extrinsics transform at infinity' \
--hardcopy ~/projects/mrcal-doc-external/figures/diff/diff-splined-opencv8.png \
--terminal 'pngcairo size 1024,768 transparent noenhanced crop font ",12"'
mrcal-show-projection-diff \
--unset key \
$D/{opencv8,splined}.cameramodel \
--title 'Diff looking at 2 models, computing extrinsics transform at infinity' \
--hardcopy ~/projects/mrcal-doc-external/figures/diff/diff-splined-opencv8.pdf \
--terminal 'pdf size 8in,6in noenhanced solid color font ",16"'
pdfcrop ~/projects/mrcal-doc-external/figures/diff/diff-splined-opencv8.pdf
#+end_src
[[file:external/figures/diff/diff-splined-opencv8.png]]
/Much/ better. As observed earlier, the Sony Alpha 7 III camera is applying some
extra image processing that's not modeled by =LENSMODEL_OPENV8=, so we see an
anomaly in the center. The models agree decently well past that, and then the
error grows quickly as we move towards the edges.
This differencing method is very powerful, and has numerous applications. For
instance:
- evaluating the manufacturing variation of different lenses
- quantifying intrinsics drift due to mechanical or thermal stresses
- testing different solution methods
- underlying a [[file:tour-cross-validation.org][cross-validation scheme]] to gauge the reliability of a calibration
result
Many of these analyses immediately raise a question: how much of a difference do
I expect to get from random noise, and how much is attributable to whatever I'm
trying to measure?
These questions can be answered conclusively by quantifying a model's projection
uncertainty, so let's talk about that now.
* Next
Now we [[file:tour-uncertainty.org][compute the projection uncertainties of the models]]
|
