1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
|
<!--
//-----------------------------------------------------------------------------
// Copyright (C) 2002-2017 Thomas S. Ullrich
//
// This file is part of "xyscan".
//
// This file may be used under the terms of the GNU General Public License.
// This project is free software; you can redistribute it and/or modify it
// under the terms of the GNU General Public License.
//
// Author: Thomas S. Ullrich
// Last update: Nov 15, 2017
//-----------------------------------------------------------------------------
-->
<html>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<title>Precision and Number Formats</title>
<body style="font-family: Arial, Helvetica, sans-serif, Geneva">
<h2>Note on Precision</h2>
<p>The precision
one can achieve depends pretty much on the quality of the plot
and of course on how careful the scan is performed. For a high
quality plot and a carefully conducted scan the precision
is realistically not better than the equivalent of approximately
one pixel. To get a feeling on the achievable precision
select <tt>View->Current Precision</tt>.
The information dialog that pops up displays the current precision
corresponding to +/- one pixel in every direction. For logarithmic
axis the uncertainties will be of course asymmetric. Although not a
perfect estimate, the values shown should give you a feeling
on the precision of the scan and how serious you should take
the obtained values. This is not applicable for heavily distorted plots.</p>
<p align="center"><img src="precision.png" alt="" width="359" border="1" /></p>
<p> </p>
<h2>Number Format and Precision</h2>
<p>Coordinates and measured/scanned values are displayed with in the 'g' format meaning either as plain decimal numbers [-]9.9 or in exponential form [-]9.9E[+|-]999, whichever is the most concise. Once the markers are all set, <em>xyscan</em> sets the "precision" of the displayed numbers to a reasonable value. In the exponential format the precision represents the number of digits after the decimal point. In the standard form the precision represents the maximum number of significant digits (trailing zeros are omitted). The number of significant digits can be changed with the <code>Edit->Significant Digits</code> menu item. If the <a href="measuring.html">Measure Tool</a> is used and no markers are set the number of significant digits is fixed at 6.</p>
<p> </p>
</body>
</html>
|
