1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
|
Installation and testing guidelines for the MUSE data reduction pipeline
================================================================================
Prerequisites
^^^^^^^^^^^^^^^
In case you do not install from the pipeline kit, you need to make sure that
some prerequisites are met:
- Download CFITSIO, wcslib, CPL, and EsoRex from
http://www.eso.org/sci/software/cpl/download.html
and install them.
Notes:
. CFITSIO needs to be configured with --enable-reentrant (for multithreading).
. CPL v6.6, configured for multithreading (the default), is required.
. CPL needs to be set up to work with wcslib (configure switch --with-wcs)
and FFTW (configure switch --with-fftw).
CFITSIO = stand-alone C library for easy FITS file access
wcslib = Astronomical World Coordinate System transformations library
FFTW = C subroutine library for computing the discrete Fourier transform
CPL = Common Pipeline Library for VLT
EsoRex = ESO Recipe Execution Tool
- libxslt should be installed, too, for the xsltproc command to work. On most
(Linux) systems it is installed by default.
- In the following we assume that you want to install the pipeline into a
directory ${MUSE_DIR} and that you have installed CPL into ${CPL_DIR},
respectively. You should replace these with real paths while carrying out the
instructions below.
- Set up esorex by editing ~/.esorex/esorex.rc (run esorex --create-config if it
doesn't exist yet) to contain at least
esorex.caller.recipe-dir=${MUSE_DIR}
Perhaps you also want
esorex.caller.msg-level=debug
to get more output while running the recipes.
- GCC 4.3 or later for parallelization, GCC 4.7 or later for full performance
You need a recent version of the GNU Compiler Collection to be able to use
OpenMP parallelization in some functions of the MUSE library. A compiler
supporting OpenMP v3.1 is needed to use the self-calibration function in
the muse_scipost recipe with full performance.
================================================================================
Installing the pipeline from a release tarball
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Unpack the sources:
tar xvjf musep-2.6.tar.bz2
- Configure the sources (and keep a log of the output):
cd musep-2.6
./configure --prefix=${MUSE_DIR} --with-cpl=${CPL_DIR} --with-cext=${CPL_DIR} 2>&1 | tee build.log
where ${CPL_DIR} is the location, CPL was installed to.
- Compile and install the sources:
make 2>&1 | tee -a build.log
make install 2>&1 | tee -a build.log
================================================================================
Installing the pipeline from SVN
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Check out the MUSE pipeline source tree from SVN:
svn co http://svnhq2.hq.eso.org/p2/trunk/Pipelines/muse muse
(Or use another tool like |git svn| or |hg svn|.)
- Configure the muse pipeline recipes code (I usually use /tmp/musetest as
${MUSE_DIR}):
cd musep
./bootstrap
(this will fail if not the right versions of autoconf, automake, aclocal, and
libtool are installed)
./configure --prefix=${MUSE_DIR} [ --with-cpl=${CPL_DIR} ] [ --with-cext=${CPL_DIR} ]
possibly add
--enable-debug (compile with debugging symbols)
--enable-strict (more warnings during compilation)
--enable-maintainer-mode (enables compilation of documentation)
Compile it:
make
make html (to generate the HTML documentation)
Install it (into ${MUSE_DIR} which I usually set to /tmp/musetest)
make install
make install-html (installs HTML documentation in
${MUSE_DIR}/share/doc/muse/html)
Note: The install targets also carry out the normal compilation if
it was not done before seperately.
================================================================================
Automated Testing
^^^^^^^^^^^^^^^^^^^
To ensure that the pipeline was installed correctly, one can run the automated
test programs.
- Unpack the current tarball of test data (named "muse_tests_YYYY-MM-DD.tar.bz2")
in the top level _destination_ directory so that the test data ends up in
${MUSE_DIR}/tests:
cd ${MUSE_DIR}
tar xvjf muse_tests_YYYY-MM-DD.tar.bz2
- Now change back to the _source_ directory of the MUSE pipeline and run the
tests (and keep a log of the output):
cd <path>/musep-2.6/ (if you install from a release tarball)
or
cd <path>/musep/ (if you install from SVN)
and then
make check 2>&1 | tee check.output
- If the output says that any tests failed, please mail check.output to the
pipeline maintainers.
- Now you can again remove data generated by this check:
rm tests/*.log tests/*.fits
================================================================================
Restrictions
^^^^^^^^^^^^^^
The following recipes work, with some restrictions:
muse_lingain (experimental, to derive gain and non-linearity correction;
needs special, long exposure sequence)
muse_bias
muse_dark
muse_flat
muse_wavecal (care has to be taken to select inputs from all lamps)
muse_lsf
muse_geometry (care needs to be taken to select a full series and
corresponding calibrations, as well as to limit the memory
usage by setting OpenMP thread limits; not strictly part of
the science pipeline)
muse_ampl (not part of the science pipeline)
muse_qi_mask (special recipe for the mask creation on the instrument
workstation; not part of the science pipeline)
muse_illum (dummy recipe, only useful for the online pipeline at Paranal)
muse_twilight
muse_create_sky
muse_standard (no support for multiple stars)
muse_astrometry (works for defined fields)
muse_scibasic
muse_scipost
muse_exp_align (works to compute offsets for most exposure sequences with
large overlaps, but not all)
muse_exp_combine (care has to be taken to limit memory usage, e.g. by
specifying wavelength ranges)
The following recipes provide functionality that is integrated into the
muse_scipost recipe. They were split off to be able to test each step
separately, but are not thoroughly tested:
muse_scipost_correct_dar
muse_scipost_raman
muse_scipost_subtract_sky
muse_scipost_subtract_sky_simple
muse_scipost_correct_rv
muse_scipost_calibrate_flux
muse_scipost_apply_astrometry
muse_scipost_combine_pixtables
muse_scipost_make_cube
================================================================================
Troubleshooting
^^^^^^^^^^^^^^^^^
- Routines in the MUSE library obey the following environment variables to
switch on experimental features:
MUSE_AIT_HACK_SLICE_NUMBER
Set this to an integer between 1 and 48 to restrict the
tracing search to this number of slices.
the tracing module will try to interpolate the missing
slices afterwards, but this is bound to give bad results
and will cause many error messages to be printed!
MUSE_BASICPROC_SKIP_GAIN_OVERRIDE
If you give a NONLINEARITY_GAIN file you can use it to
only correct for nonlinearities and skip applying the
gain by setting this variable to a positive integer.
MUSE_BASICPROC_SKIP_NONLIN_CORR
If you give a NONLINEARITY_GAIN file you can use it to
only override the gain and skip applying the nonlinearity
correction by setting this to a positive integer.
MUSE_TWILIGHT_SCALES
A comma-delimited string of 24 numbers which are
interpreted as relative scales for each IFU, before
constructing the DATACUBE_SKYFLAT.
MUSE_TWILIGHT_SKIP Set this to a filename of a DATACUBE_SKYFLAT as saved by
the muse_twilight recipe, to start from that instead of
spending a lot of time to reconstruct it again; to be
used for debugging!
MUSE_GEOMETRY_SKIP Set this to a positive integer to skip part of the
loading/processing. 1 means to skip raw image processing
and start from reduced data (which has to be present in the
current dir!). 2 means to skip both raw image processing
and the spot measurements (needs first reduced file and
the spots tables in the current dir!).
MUSE_GEOMETRY_NO_INVERSION
Set this to a positive integer to deactivate after-the-fact
flipping of the slices along the vertical direction.
MUSE_GEOMETRY_MASK_ROTATION
If the multi-pinhole mask was aligned along a diagonal,
set this variable to the angle of this rotation (so
either 11.7675 or 22.6185 degrees), so that such global
misalignment can be taken into account when comparing to
the expected mask hole distances.
MUSE_GEOMETRY_PINHOLE_DY
Use the value of this variable as the pinhole distance in
vertical direction [mm]. This overrides the internally
computed value (normally accurate to about 0.0001 mm).
MUSE_GEOMETRY_STD_GAP
If this variable is set to a positive integer, the
horizontal geometrical computation will use an older
way to compute the gaps.
MUSE_GEOMETRY_HORI_OFFSETS
Set this to a comma-delimited list of floating point
values to apply additional offsets to the internal
horizontal IFU location refinement. The values are
interpreted as horizontal shifts in units of pixels.
MUSE_DAR_CORRECT_METHOD
Switch methods to correct differential atmopheric
refraction (DAR). Four values are valid:
- "Filippenko": use the DAR correction method from
Filippenko (1982); this is the default
- "Owens": use the DAR correction method from Owens
(1967) which is also used by Marchetti (2001)
- "Edlen": use the refractive index calculation from
Edlen (1966) and Birch & Downs (1993)
- "Ciddor": use the refractive index calculation from
Ciddor (1966); this is supposed to give the most
accurate results
MUSE_PIXTABLE_SAVE_AS_TABLE
When this variable is set to a positive integer, pixel
tables are saved as FITS binary tables instead of multi-
extension FITS images. On most hardware it should be
faster to use the default. (Loading any pixel table in a
later processing stage detects the file format and acts
accordingly.)
MUSE_COLLAPSE_PIXTABLE
set this to a positive value to get the collapsed image
of the field of view created from data before resampling;
per filter this takes as long as the resampling to cube;
This only works, if the output format is not Euro3D.
MUSE_COLLAPSE_USE_VARIANCE
set this to a positive value to weight datapoints using
inverse variance (to suppress low S/N regions) while
collapsing data to an image. This applies both in normal
operation and when MUSE_COLLAPSE_PIXTABLE is set.
MUSE_CPL_ERRORSTATE_NDUMP
The MUSE recipes output an error summary at the end of
the processing. By default this lists the last 20 errors
that happened during processing. Set this variable to a
positive value to output any other number.
(Note that even when setting this to a high value, most
of the errors will display as 'The actual error was
lost', since CPL only keeps a certain number of errors in
its history.)
MUSE_EXPERT_USER Set this environment variable if you are familiar with
the code of the MUSE pipeline and are happy to check the
output/log for WARNINGs and ERRORs, can deal with partly
filled products, occasional crashes, and unexpected
behavior.
- Some routines in the MUSE library obey the following environment variables to
enable extra debugging output:
MUSE_DEBUG_QUADRANTS
switch on additional debugging messages about the
location of quadrants on the CCD and trimmed regions
MUSE_DEBUG_TRACE switch on additional debugging messages in tracing
MUSE_DEBUG_WAVECAL set this to positive integers to get subsequently more
debug output, a (max) value of 3 or higher will produce
an ASCII output file "MUSE_DEBUG_WAVE_LINES-nn.ascii"
(where nn is the IFU number) with all detected lines and
their associated wavelengths
MUSE_DEBUG_IGNORE_READOUT and MUSE_DEBUG_IGNORE_INSMODE
These variables can be used to override pipeline guards
against applying unsuitable calibrations. Specifically,
if the first variable is set to a positive integer,
a master bias of a different read-out mode can be applied
to another raw data set. The second variable can be used
to apply a flat-field to a dataset of a different
instrument mode.
MUSE_DEBUG_LSF_FIT set this to a positive integer to get lots of debugging
output for every iteration of the LSF fit procedure
MUSE_DEBUG_AUTOCALIB
set this variable, and the pipeline saves an extra column
"MASK" with the pixel table during autocalibration; this
also causes the autocalibration to output a lot more
debug-level info and statistics for all slices of all
IFUs for all lambda ranges; otherwise only progress
messages are given.
MUSE_DEBUG_SKY set this to a positive integer to more debugging output
from sky subtraction
MUSE_PLOT_TRACE will try to use gnuplot to visualize the detected slices
and relevant limits (if value is 1)
MUSE_DEBUG_DCR switch on additional debugging messages in the cosmic ray
detection following the DCR approach (values of 1, 2, 3,
and 4 result in subsequently more debug output), some may
also write extra files with debugging output
MUSE_DEBUG_FLUX set this to to a positive integer to get debug output and
files while carrying out flux calibration:
>= 1: files flux__sens_*.ascii containing different
sensitivity calculation stages
>= 2: files flux__cube.fits, flux__intimage.fits, and
possiblty flux__tellregions.fits containing the
reconstructed cube, a special image file with
measurements of integrated flux, and the table
of the builtin telluric regions (if used)
> 2: processed pixel table used to reconstruct the cube
MUSE_DEBUG_PIXTABLE_LIMITS
if set to a non-zero value, output coordinate limits of a
pixel table after computing them
MUSE_DEBUG_GEO_VERIFY_DY
if set to a positive integer, verify vertical pinhole
mask distances using a set of output files
MUSE_DEBUG_GEO_VERTICAL
if set to a positive integer, output extra debugging
about the vertical distances between slices in each stack
MUSE_DEBUG_MEMORY_PROGRAM
set to a executable name for which to print memory
information in selected parts of the pipeline
MUSE_DEBUG_WCS if set to a positive integer, output debugging info when
creating new WCS info from a FITS header and carrying out
the astrometric calibration. In the latter case, extra
files are saved for the following values:
>= 2: files wcs__cube.fits, wcs__image_median.fits, and
wcs__detections.fits, containing the cube, a
median detection image, and a table of detected
sources for the astrometric calibration, as well
as ASCII files wcs__{detections,references}.ascii
containing tables of detected and reference
sources
> 2: processed pixel table used to reconstruct the cube
MUSE_DEBUG_GRID_CONVERSION
set to a non-zero value in builds with --enable-debug
will output information about which pixtable row ends up
where in the 3D grid when resampling to a datacube
MUSE_DEBUG_NEAREST set to a non-zero value in builds with --enable-debug
to get information about nearest-neighbor resampling
MUSE_DEBUG_WEIGHTED
The following non-zero values cause special debug output
in the resampling routines in builds with --enable-debug:
1: display sums 3D case
2: display contributing pixels 3D case
(needs X, Y, Z coordinates, see below)
4: display individual pixels 3D case
8: detailed info about weights of all 3D case
contributing pixels
128: display contributing pixels 2D case
(needs X, Z coordinates, see below)
256: display individual pixels 2D case
The values can be combined, i.e. a value of 3 means to
output both sums and contributing pixels.
MUSE_DEBUG_WEIGHTED_X
MUSE_DEBUG_WEIGHTED_Y
MUSE_DEBUG_WEIGHTED_Z
when all of these variables are set to a positive value,
then they are used to display pixtable entries
contributing to the output pixel at this grid point in
the output cube (values starting at 1).
MUSE_DEBUG_WEIGHTED_GRID
Set this to a valid filename to produce an ASCII file
of the constructed grid. The columns in that table are
the grid cell numbers (starting from 1) and the three
coordinates of the cell centers.
MUSE_DEBUG_WEIGHT_CUBE
set this to a valid filename to produce a weight cube
(analogous to a weight map in imaging) in addition to a
final datacube when carrying out cube reconstruction
using a weighted resampling method
MUSE_DEBUG_CRREJECT
The following non-zero values cause special debug output
in the cosmic-ray rejection during final resampling, in
builds with --enable-debug:
1: display pixel table entry for rejected cosmic ray
2: display statistics of surrounding pixels
4: debug filling of statistics object
All need X, Y, Z coordinates, see below.
MUSE_DEBUG_CRREJECT_X
MUSE_DEBUG_CRREJECT_Y
MUSE_DEBUG_CRREJECT_Z
Debug coordinates, similar to MUSE_DEBUG_WEIGHTED_*.
If you have problems with some functions, switch them on to see more, possibly
helpful output. If you cannot solve a problem, ask for help. If you are part
of the MUSE consortium, use the CRAL gitlab ticket system at
https://git-cral.univ-lyon1.fr/MUSE/DRS/issues
(An account is required.)
otherwise write to usd-help@eso.org.
|
